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Foreword 



An encouraging trend in the field of artificial intelligence in the past few years 
has been its growing interaction with the area of operations research. AI has 
traditionally concentrated on problems of logical inference and satisfiability, 
or in other words, Boolean feasibility problems. OR, by contrast, has mainly 
focused on problems of linear optimization. Many significant real world tasks 
share aspects of both kinds of problems, and there is therefore much inter- 
est in integrating and expanding the techniques that have been developed in 
each field. This monograph by Dr. Joachim Paul Walser is thus particularly 
timely and significant. He deve lopes new algorithms and systems for applying 
discrete local search techniques that were originally developed for Boolean 
feasibility problems to a broad class of integer linear optimization problems. 
He demonstrates that his general, domain-independent solver can be com- 
petitive with specialized algorithms on hard realistic problems, and can often 
far outperform other state-of-the-art domain-independent solvers. Through- 
out the monograph Dr. Walser draws connections to classic techniques from 
OR and AI, and demonstrates how different approaches (such as local search 
and linear relaxations) can be combined to solve relevant problems in integer 
programming. 

The contributions of this monograph can be placed in two groups: First, 
there is the specific algorithmic work, including extending the “walksat” al- 
gorithm from Boolean to integer constraints and creating a new local search 
strategy for over-constrained problems. The resulting Wsat(oip) system rep- 
resents the distillation of a careful and deep search through the space of 
possible designs, and its elegance, and empirical success are remarkable. To 
take one specific example: walksat’s performance is known to degrade in the 
presence of constraints containing many variables. However, Wsat(oip) can 
efficiently handle the very long constraints (over 300 variables per constraint) 
that arise in many real-world domains. 

Second, the book presents a series of well-chosen empirical evaluations. 
The different cases represent a spectrum of different kinds of problems: feasi- 
bility versus optimization, loosely-constrained versus highly constrained, and 
0/1 versus integer valued variables. Some of the most significant results Dr. 
Walser reports are on problems of capacitated production planning. These 
are very large, hard industrial problems, which are (unfortunately) all too 
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rare in academic research. Dr. Walser demonstrates that Wsat(oip) can 
find solutions that are much closer to optimal than could be found by any 
competing approach. The empirical evaluation shows, for the first time, that 
a completely general local search engine (as opposed to domain-specific local 
search algorithms) can efficiently find optimal or near-optimal solutions to 
a broad range of real-world problems, and thus complement established sys- 
tematic problem-solving frameworks such as integer prorgramming branch- 
and-bound. 

In short, this material is mandatory reading for researchers in AI who are 
seriously concerned with solving hard combinatorial search and optimization 
problems, as well as those researchers in OR who want to see the best that 
AI has to offer. The clarity and breadth of the presentation also makes this 
book an excellent choice for reading material in a graduate or advanced un- 
dergraduate seminar in either AI or OR. 

February 1999 Henry A. Kautz 

Florham Park, New Jersey, USA 




Preface 



Software to support complex planning decisions is becoming a vital factor 
for competitiveness, driven by the increasing availability of organizational 
data in modern enterprise information systems. Decision support software 
can reduce manufacturing costs, increase organizational efficiency, and deliver 
solutions to complex resource allocation problems - by building on effective 
optimization algorithms. 

Integer optimization covers a variety of practically important optimization 
problems, including production planning, timetabling, VLSI circuit design, 
network design, logistics, or sports scheduling. The goal of integer optimiza- 
tion is to solve a system of constraints over many discrete variables, and to 
find solutions that are ‘good’ in terms of given optimization criteria. While 
fast general-purpose algorithms for solving large systems of linear inequalities 
over continuous variables are well-established (linear programming), integer 
optimization problems which include discrete decisions pose a difficult chal- 
lenge to algorithmics. Yet, discrete decisions (“the truck leaves either today 
or tomorrow” ) are a critical part of most real-world planning and scheduling 
scenarios. 

In the recent past, the field of integer and combinatorial optimization has 
gained momentum, and amongst the many new algorithms, heuristics have 
taken a leading role in finding near-optimal solutions to specific optimiza- 
tion tasks. The success of special-purpose heuristics is mainly due to their 
effectiveness for large practical problems - even if they come with no theo- 
retical guarantee of optimality. The drawback of special-purpose algorithms, 
however, is their limited applicability. As a result, many practical optimiza- 
tion problems are still attacked in an ad-hoc fashion since practitioners often 
lack the time and expertise to research and develop effective special-purpose 
algorithms for the diverse optimization problems that arise. 

This monograph explores a new domain-independent approach to integer 
optimization, which, unlike traditional strategies for integer optimization, is 
based on local search. It develops the central ideas and strategies of inte- 
ger local search and describes possible combinations with classical methods, 
such as linear programming. In a number of case studies, it demonstrates 
the surprising effectiveness of the approach for a variety of realistic discrete 
optimization problems. 
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Preface 



Like traditional strategies for integer linear programming, integer local 
search operates on an abstract model of the problem to be solved and can 
thereby exploit the underlying commonalities shared by many real-world 
problems. As a result, solvers based on the technology described here can be 
combined with existing off-the-shelf modeling languages for integer program- 
ming and can be applied to many integer optimization problems without the 
need of code implementation. We investigate the potential of integer local 
search for various domains (time tabling, sports scheduling, radar surveil- 
lance, course assignment, and capacitated production planning) and com- 
pare the experimental results to state-of-the-art integer programming and 
constraint programming approaches. 

This book is written for researchers and practitioners in the area of com- 
binatorial optimization from artificial intelligence and operations research. 
Developers with an interest in the design of optimization algorithms will 
benefit from the detailed description of new local search strategies. Practi- 
tioners in the field can obtain insights into modeling issues and learn about 
the capabilities of integer local search, which often surpasses state-of-the-art 
IP solvers for the domains under investigation. 

This book is organized as follows. Chapter 1 introduces the context in 
which this work is situated, integer optimization, heuristics, and local search. 
It provides a high-level description of the basic strategy of integer local search 
and the underlying representation of over-constrained integer programs. It 
also outlines the experimental results from the application case studies. Chap- 
ter 2 briefly introduces important general frameworks for combinatorial op- 
timization and their terminology, i. e. integer linear programming branch- 
and-bound, finite domain constraint programming, and local search. It also 
discusses complementary search relaxations as a new characteristic to classify 
optimization methods. 

Chapter 3 contains the technical contributions, over-constrained integer 
programs and presents an in-depth description of the integer local search 
method Wsat(oip). It also discusses several possible combinations with lin- 
ear programming and illustrates different variations of the Wsat(oip) strat- 
egy with graphical examples. Chapter 4 describes the case study methodol- 
ogy. It discusses criteria of success for practical optimization methods and 
motivates the experimental design and the problem selection. 

The remainder of the text describes the case studies, each chapter focusing 
on a particular problem type, and providing evidence that important criteria 
of success are met by integer local search. Chapter 5 describes applications in 
time-tabling and sports scheduling. Chapter 6 radar surveillance and course 
assignment, and Chapter 7 presents an application to capacitated production 
planning. Chapter 8 finally discusses limitations and extensions of the current 
methods and concludes with suggestions for future work. 
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1. Introduction 



“Integer programming has gone through many phases in the last three 
decades, spurred by the recognition that its domain encompasses a wide 
range of important and challenging practical applications." 

Fred Glover in [56], 1986 



Integer and combinatorial optimization problems arise when a large num- 
ber of discrete organizational decisions have to be made, subject to con- 
straints and optimization criteria. This monograph describes and investigates 
new domain-independent local search strategies for linear integer optimiza- 
tion. This chapter briefly introduces integer optimization and heuristics, and 
presents an outline of integer local search, the approach to integer optimiza- 
tion that is the subject of this book. Integer local search generalizes local 
search for propositional satisfiability to linear integer optimization. Research 
in this area is situated in the interface between artificial intelligence and op- 
erations research. Since the two fields have been relatively separated in their 
past, terminology conflicts occasionally arise that we will attempt to point 
out. 



1.1 Integer Optimization and Henristics 

A major challenge in algorithmics to date is to devise efficient and robust 
methods for combinatorial optimization problems. Combinatorial optimiza- 
tion is the problem of solving a system of constraints over many discrete 
variables and finding solutions that maximize or minimize some optimiza- 
tion criteria [117, 114]. The complexity of many interesting combinatorial 
optimization problems is known to be AP-hard.^ 

Combinatorial optimization problems vary largely, prototypical examples 
ranging from time-tabling over machine-scheduling to resource allocation. A 
problem class that captures a wide range of practically important problems 



^ A search problem X is AP-hard if for some TVP-complete decision problem Y 
there is a polynomial-time reduction from Y to X [87]. 
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is the integer linear programming problem (ILP).^ An ILP consists of a set of 
linear inequalities (constraints) over integer variables, and a linear objective 
function, and is usually defined [114] as: 

(ILP) min { cx : Ax > b, x € Z” }, 

where Z" is the set of nonnegative integral n-dimensional vectors and x = 
(a;i,..., Xn) are the variables. An instance of the problem is specified by 
the data (c, A, b), with b and c n-vectors and A an m x n matrix, and all 
numbers are rational (note that an equality constraint can be represented 
by two inequalities). If all variables are binary (0-1), the problem is also 
called 0-1 ILP. Throughout this book, we assume the optimization objective 
to be minimizing, and focus on integer programming problems with linear 
constraints and objective functions. A variable assignment that meets all 
constraints is called a feasible solution. 

A wide variety of methods have been developed for solving ILP problems 
and are the subject of ongoing research in mathematical programming. When 
the integrality restrictions (x G Z") of an ILP are relaxed, one obtains the 
well-studied linear programming problem (LP) for which polynomial algo- 
rithms are known [92] and efficient implementations exist. ILP methods can 
make use of LP relaxations in many ways, for instance for lower bounding or 
feasibility testing, and most integer programming (IP) frameworks are based 
on iteratively solving LP relaxations, e. g. branch-and-bound or cutting-plane 
algorithms [114]. In summary, ILP provides a good starting point as repre- 
sentation for general-purpose optimization methods. 

Domain- Specific vs. Domain-Independent Techniques. 

There are two orientations of research on optimization algorithms. First, 
specialized techniques that excel in solving narrow classes of optimization 
problems for which maximal quality is crucial and development times can 
be neglected. Such techniques often solve sub-problems of the general ILP 
problem, for example the Set-Covering Problem (SCP). 

Let M = {!,..., m} be a finite set and let {Mj} for j G I = {1, . . . , n} 
be a given collection of subsets of M. We say that F C I is a cover of M 
if Uj6F ^3 ~ set-covering problem, Cj is a cost associated with 

Mj, and we seek a minimum-cost cover [114]. SCP can be formulated as an 
integer linear program using 0-1 variables Xj with Xj = 1 if and only if j is 
in the cover. ^ The ILP formulation of SCP is: 

n 

(SCP) min { cx : ^ ^ijXj > 1, i = 1, . . . ,m, x G {0, 1}" }, 
i=i 



^ The term ‘programming’ dates back to the 1940s, when Dantzig described the 
simplex method for linear programming. ‘Programming’ was a military term 
that, at the time, referred to planning and scheduling of logistics. 

® Xj G {0, 1} can be constrained through inequalities Xj > 0, —Xj > —1. 
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where c is an n- vector, (oy) is a 0-1 matrix, and the variables are binary 
(0-1). Domain-specific strategies for the set-covering problem are restricted 
to problems of the form (SCP) as specified. Incorporating other constraints 
typically requires adjusting the algorithms or replacing the strategies alto- 
gether. 

In contrast, domain-independent techniques strive to be flexible and ap- 
plicable to a wider range of practical problems without the need of designing 
strategies on a class-by-class basis [59]. Such techniques work from a model 
of a given problem instance (a representation in a suitable constraint class). 
They are of practical importance because practitioners often lack the nec- 
essary time and expertise to research and develop effective special-purpose 
algorithms. Further, in real environments, flexibility is often critical to re- 
spond to rapidly changing requirements. 

Research on domain-independent techniques for combinatorial optimiza- 
tion has given rise to general-purpose tools in integer programming, such as 
a variety of branch-and-bound solvers (e. g. Cplex, LINDO, XPRESSMP, 
MINTO to name but a few). More recently. Constraint Programming (CP) 
systems have entered the picture that support rapid development of domain- 
specific methods and incorporate an increasing variety of techniques for con- 
straint propagation and search (e. g. CHIP [42], Oz [137], ILOG solver [120]). 

Heuristics and Local Search. Optimization methods can either be exact 
or approximate. While exact methods perform a systematic search for op- 
timal solutions, approximate methods provide no theoretical guarantee for 
finding optimal or even feasible solutions. In operations research, approxi- 
mate methods are commonly termed heuristics, and we will stay with this 
usage. ^ Heuristics concentrate on finding near-optimal solutions quickly, and 
have received much interest in recent years due to their practical success 
[126, 123, 1, 61]. 

An important class of heuristics is local search [1] which has a long history 
for combinatorial optimization and dates back to methods for the traveling 
salesman problem in the 1950s and 1960s [20, 37, 102]. The key idea behind 
local search is to start from a solution and iteratively perform changes to 
improve it. There are many variations of local search methods, sharing the 
common notion of local moves which are transitions in the space of (feasible 
and possibly infeasible) solutions, typically according to a strategy that works 
by improving the local gradient of a measure of the solution quality (a strategy 
called hillclimhing) . Many variants of local search exist that can be applied 
to combinatorial optimization problems, prominent examples being simulated 
annealing [96] tabu search [57, 61], genetic algorithms [113, 62] or the greedy 
randomized adaptive search procedure (GRASP) [128]. 

In artificial intelligence, local search strategies have recently seen much 
success for model finding in propositional satisfiability [135, 64, 111, 53, 134] 

^ Note that in artificial intelligence, the term ‘heuristic’ commonly refers to a 
‘rule-of-thumb’ decision strategy of an algorithm. 
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and a variety of applications to combinatorial problems have been reported 
[132, 35, 128, 54, 93]. Local search strategies of this kind are also called itera- 
tive repair [108, 155, 135]: Given a problem that is stated in terms of variables 
and constraints, one first generates some initial assignment of all variables, 
normally violating a number of constraints. Subsequently, variable values are 
changed in order to reduce the number of conflicts with the constraints, i.e. 
in order to repair the current variable assignment. 

Heuristics for Integer Optimization. Most heuristics for integer opti- 
mization are dedicated to a specific problem (like set-covering or job-shop 
scheduling) and often excel in terms of the quality of solutions found and 
efficiency. Perhaps surprisingly, only few efforts have been made to devise 
heuristics that target a wider range of combinatorial optimization problems 
and operate on problem representations using constraints. 

Recently, several general-purpose heuristics have been described which 
aim at solving general ILP problems (some being extensions of the pioneer- 
ing work by Balas and Martin [9]). These heuristics are of two types, (i) 
approaches which relax the integrality constraints and primarily operate on 
continuous variables [9, 3, 104, 59, 61] (e.g. by solving the linear program 
followed by special pivot moves), and (ii) local search methods in which the 
local moves are performed directly in the space of integer solutions, such as 
simulated annealing [33, 4] and stochastic local search [149, 150].® 

The methods presented in this monograph are of the second type and 
arise from generalizing successful strategies of local search for propositional 
satisfiability [149, 150]. We will refer to these methods as integer local search. 

Local Search for Propositional Satisfiability. A problem of much in- 
terest in computer science is the propositional satisfiability problem (SAT). 
Let y be a finite set of 0-1 (Boolean) variables. An assignment for P is a 
mapping from V to {0, 1}. A literal is either a variable v or its negation v 
(U = 1 iff u = 0). A clause is a set of literals, and is satisfied by a given 
assignment A if at least one of its literals is assigned to 1. A set of clauses C 
is interpreted in conjunctive normal form (CNF): An assignment A satisfies 
C if and only if all clauses in C are satisfied under A. 

(SAT) Given a set V of variables and a set C of clauses 
over V, is there an assignment for V that satisfies all 
clauses in Cl 

We observe that SAT is also a special case of 0-1 ILP (with objective 
function c = 0), since each clause can be translated to a linear inequality: 
For instance, the clause {x, y} can be translated to (1 — a;) -I- y > 1. The 0-1 
ILP problem is more general than SAT because it allows for arbitrary right- 
hand-sides (b) and coefficients (A), and because SAT is a decision problem, 
i.e. there is no explicit representation of an objective function in SAT. 

[9, 3, 104] actually contain phases of type (ii). 
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The Walksat Strategy. A number of efficient local search strategies have been 
developed for SAT in recent years, one of the most successful ones being the 
Walksat procedure by Selman, Kautz, and Cohen [134, 106]. To find solutions 
to a set C, the basic Walksat strategy performs a greedy local search equipped 
with a ‘noise’ strategy: Initially, all variables are assigned a random value 
from {0, 1}. While some of the clauses will be satisfied, others are violated. 
To seek an assignment that satisfies all clauses, the method iteratively selects 
a violated clause c G C, and from c selects a variable such that changing its 
value yields the largest increase in the total number of satisfied clauses. If no 
variable exists that improves the total number of satisfied clauses, a variable 
from c is selected at random according to some detailed scheme. Such variable 
changes (flips) are repeated a fixed maximal number of iterations after which 
a restart takes place. If no satisfying assignment is found after a fixed number 
of restarts, the procedure is terminated unsuccessfully. 



1.2 Integer Local Search 

This monograph is concerned with local search strategies for integer optimiza- 
tion. It describes, discusses and empirically analyzes Wsat(oip), a domain- 
independent method that generalizes local search for propositional satisfia- 
bility (the Walksat strategy) to integer optimization and integer constraint 
solving. 

Over-constrained Integer Programs. Many integer optimization prob- 
lems have no concise SAT encoding, and hence SAT local search algorithms 
cannot be applied. In order to generalize SAT local search to integer opti- 
mization, we introduce an extension of SAT to a constraint system called 
over-constrained integer programs (OIPs). Extending the repair strategy of 
Walksat to OIP optimization will then be a natural step. 

An OIP consists of hard and soft inequality constraints, wherein the opti- 
mization objectives are represented by the soft constraints. If all inequalities 
are linear, the OIP problem can be formulated in matrix notation as 

Ax > b, Cx < d (soft), X G D, 

where A and C are mx n-matrices, b, d are m- vectors, and x = (a;i, . . . , x„) is 
the variable vector, ranging over positive finite domains Xi € Di. A variable 
assignment that satisfies all hard constraints is called a feasible solution. 
Given a tuple (A, b, C, d, D), the OIP minimization problem is 

(OIP) min{||Cx — djj : Ax > b, x G D }, ||v|| ^ max(0, Uj), 

i 

wherein the objective is to find a feasible solution with minimal soft con- 
straint violation. In ||.||, the contribution of each violated soft constraint to 
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the overall objective is its degree of violation. As will be shown, OIPs are a 
special case of integer linear programs because each soft constraint encodes 
a piecewise-linear convex objective function. The reduction enables effective 
combinations with linear programming for lower bounding, initialization by 
rounding, search space reduction, and feasibility testing. 

While ILP encodes the optimization objectives using a monolithic objec- 
tive function, OIP uses many competing soft-constraints. OIPs are a natural 
representation to generalize iterative repair strategies like Walksat: Only vi- 
olated constraints need to be repaired, and no principal distinction is drawn 
between repairing hard and soft constraints, thereby seeking solutions that 
are both feasible and near-optimal. OIP is similar enough to ILP to make use 
of algebraic modeling languages like AMPL [48] as front-end to an integer 
local search solver. 

Unlike methods that rely on properties of the linear relaxation, note that 
integer local search is not limited to inequality and equation constraints but 
can be extended to other types of constraints, for example disequality con- 
straints {x yf y) or symbolic constraints {all-different{xi, . . . ,x„)). 

The Wsat(oip) Strategy. Wsat(oip) is now a natural generalization of 
Walksat. It performs local moves in the space of feasible and infeasible so- 
lutions by repairing constraint violations. Iteratively, Wsat(oip) changes a 
variable value as follows. First, select a violated constraint c: if only hard or 
only soft constraints are violated, select c at random. If hard and soft con- 
straints are violated, with some probability Phard select a random violated 
hard constraint and with 1 — Phard ^ random violated soft constraint. The 
Phard parameter controls how quickly the search is driven into the feasible 
region of the search space. 

Next, to decide which variable from c to change, it no longer suffices to 
reduce the number of violated constraints as Walksat does. Instead, a score 
is maintained that accounts for both the soft constraint violation and the 
degree of infeasibility of an assignment (using ||.|| above for both soft and 
hard constraints). For 0-1 variables, a variable change amounts to flipping 
the value. To extend the principle to finite domain variables, a new class 
of local moves is introduced that changes a variable value to a smaller or 
greater values nearby. Like variable flips, such trigger moves are also induced 
by violated linear inequalities and only repairs in c are considered that reduce 
the violation of c. 

Among the possible repairs, the strategy greedily chooses the one that 
most improves the overall score, but occasionally resorts to random moves if 
no improvement is possible. Following Walksat, Wsat(oip) is stochastic local 
search and inherits Walksat’s noise strategy. To improve the performance for 
realistic problems, the method is further extended with principles from tabu 
search [57, 58] by keeping a history of past moves. History information can be 
used to diversify the search process. For instance, variables can have a tabu 
status that disallows changes that would undo a recent change (tabu status 
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can be overridden by aspiration criteria). Finally, detailed tie breaking rules 
[53] are built into the strategy that are based on the history of the search and 
decide which move to perform when several variables have an equal score. 



1.3 Experimental Results 

A substantial part of this book is concerned with an empirical investigation 
of the Wsat(oip) method for practical optimization problems. The problems 
under consideration are a resource allocation problem termed ‘the Progres- 
sive Party Problem’ (PPP) [136], the scheduling of a basketball season (ACC) 
[115], radar surveillance (RS) [24], course assignment (CA), and capacitated 
production planning (CPP) [150]. The problem instances stem from real ap- 
plications (ACC, CA, CPP) or contain realistic structure (PPP, RS). Each 
class contains large-scale instances, involving several thousand variables and 
constraints. 

We compare the results of integer local search Wsat(oip) to approaches 
from the recent literature (ACC [115, 69], PPP [136, 76]), and to some of 
the best state-of-the-art methods for the respective problems. In particular, 
we compare the results to IP branch-and-bound (Cplex 5.0 [80], an efficient 
general-purpose mixed integer programming solver), several constraint pro- 
gramming approaches and a previous general-purpose simulated annealing 
method for 0-1 ILPs (gpsiman) [33]. 

The problems fall into three broad classes, grouped in three experimental 
chapters: (i) tightly constrained 0-1 integer constraint problems (PPP, ACC), 
(ii) 0-1 integer optimization problems (RS, CA), and (iii) integer optimization 
problems with finite domain variables (CPP). 

Because the problems vary remarkably, the experimental results demon- 
strate the general-purpose nature of the method. Further, the different types 
of problems will shed light on different performance aspects: Scaling of run- 
time with increasing size, scaling with increasing constrainedness, and resid- 
ual robustness (performance variation on a distribution of similar instances) . 
The experimental results can be summarized according to the three problem 
types. 

(i) 0-1 integer constraint problems from time-tabling (PPP, ACC). Even 
though such feasibility problems can be formulated as ILPs, these en- 
codings pose a difficult challenge for integer programming strategies 
[136, 76, 115, 149]. Our case studies demonstrate that integer local search 
can solve such problems directly from a 0-1 representation. 

For PPP, a constraint programming approach is known [136] to solve 
individual instances of the problem, while the problem appears to be 
beyond the size limitations of integer programming branch-and-bound 
[136, 76]. Our experiments show that the core problem can be solved 
more efficiently using Wsat(oip). Further, when slight variations of the 
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instance given in [136] are considered, we find that local search is robust 
with respect to the modifications, while we were not able to find a CP 
enumeration strategy to solve all test problems. 

The second problem, ACC basketball scheduling, was presented and 
solved in a multi-stage approach in [115]. The approach uses IP branch- 
and-bound and explicit enumeration in four separate stages, with a per- 
formance of roughly 24h to generate a set of feasible schedules. A more 
efficient CP strategy is also known [69]. We present a 0-1 linear integer 
encoding of the problem that does not separate the problem into stages. 
Given this monolithic 0-1 representation, we study the performance of 
integer local search with increasing problem constrainedness. Our exper- 
iments show that for loosely constrained instances, integer local search 
finds solutions in seconds. As the constraints are tightened, runtimes tend 
to increase moderately. When the entire constraints from [115, 141] are 
encoded, we show that Wsat(oip) still finds solutions in 30 minutes, 
despite only 87 solutions remain! In experiments with IP branch-and- 
bound we find that Cplex manages to solve only the loosely constrained 
instances within reasonable runtime. 

(ii) 0-1 integer optimization problems (RS / CA — extended set covering 
/ generalized assignment) are classical domains of IP branch-and-bound 
methods, and pose a difficult challenge for constraint programming ap- 
proaches. 

On these problems, the experiments show that Wsat(oip) and Cplex 
efficiently find solutions of similar quality for small problems. As the 
problem size increases, the runtime of Wsat(oip) scales gracefully. In- 
teger local search therefore outperforms IP branch-and-bound on large 
problems by orders of magnitude. 

In one series of experiments on (RS), we further find that the OIP rep- 
resentation (soft constraints) is critical for Wsat(oip) to achieve the 
above reported performance. The results show that the OIP representa- 
tion captures structural aspects that can be exploited by integer local 
search, leading to a promising primal heuristic for large-scale problems. 

(iii) OIPs with finite domain variables. The last experimental chapter stud- 
ies a real production planning problem (CPP) from the process industry 
(manufacturing of chemicals, food, plastics, etc.), technically termed ‘ca- 
pacitated lot sizing’. 

Because the problem requires discrete lot-sizes, domain-specific methods 
from the literature are not directly applicable. We therefore approach the 
problem with Wsat(oip), and empirically compare the results to a mixed 
integer programming branch-and-bound approach (again using Cplex) 
on real problem data. We find that integer local search is considerably 
more robust than MIP branch-and-bound in finding feasible solutions 
in limited time, in particular as the capacity constraints are tightened. 
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With respect to production cost, both methods find solutions of similar 

quality. 

With respect to parameters for Wsat(oip) we find that a set of standard 
parameters usually performs well on the given domains without much manual 
fine-tuning, and little further customization was necessary, with exception 
of constraint- weights (used for CPP), and redundant constraints (used for 
ACC). 

GPSIMAN. Where applicable, we additionally perform experiments with GP- 
SIMAN, a general-purpose simulated annealing method [33]. GPSIMAN was 
not able to find solutions to the 0-1 constraint problems, even at the lowest 
tightness levels within 12 hours. For the optimization problems under con- 
sideration, GPSIMAN was able to solve instances of only one class (RS), but 
did not succeed in solving the larger instances satisfactorily. 

Limitations and Scope. The computational study of integer local search 
described here contains several limitations. First, throughout our experi- 
ments, we assume pure integer optimization problems. Industrial problems, 
however, often contain a continuous component and require handling mixed 
integer programming problems, which are not addressed in the framework of 
integer local search as presented in this monograph. 

Second, our case studies do not investigate problems that contain a strong 
intricate solution structure, like traveling salesman problems or job-shop 
scheduling. Encodings of these problems into ILP are typically large and 
the solution structure is difficult to maintain by local moves that change sin- 
gle variable- values, as performed by Wsat(oip). For such problems, more 
informed local moves need to be considered. Another general and effective 
strategy for such problems is “Abstract Local Search” by Crawford, Dalai 
and Walser [36] . In Abstract Local Search, a greedy constructive heuristic is 
combined with local moves in an abstract search space. 

Third, with exception to one problem class (CPP), we observe that the 
instances under consideration contain mainly 0-1 coefficients. In fact, for the 
basic version of Wsat(oip), strongly differing coefficients are problematic 
because the score gradient favors variables that appear with large coefficients. 
An extension of the basic scoring scheme will be discussed that addresses 
larger coefficients. 



1.4 Research Contributions 

Although some earlier research papers have described basic strategies and 
case studies of Wsat(oip) [149, 150] and several related general-purpose 
heuristics have been reported [33, 116, 4, 5], this monograph represents the 
first in-depth treatment of integer local search. It advances the state-of-the- 
art of heuristics for integer optimization by the following contributions. 
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Generalization of SAT local search to linear integer optimization over binary 
(0-1) and finite domain variables. 

Development of over-constrained integer programs (OIP) as a basis for ap- 
plying iterative repair to integer optimization. Reduction from OIP to 
ILP in order to permit combinations with linear programming. 

Engineering of an effective local search strategy for over-constrained inte- 
ger programs (Wsat(oip)) including local moves for ‘triggering’ finite- 
domain variable values. 

Description of strategy enhancements used by an efficient solver, Wsat(oip), 
which interfaces with AMPL and Cplex, and which provides the first 
local search solver for integer optimization to interface with an algebraic 
modeling language. 

Development of the notion of integer local search and its characterization in 
terms of search relaxations. 

Empirical study of the approach demonstrating its flexibility, efficiency and 
graceful scaling for a variety of hard realistic integer optimization / inte- 
ger constraint problems. Within the empirical study, the original results 
include 

— evidence that large or tightly constrained problems beyond the limita- 
tions of ILP branch-and-bound can be solved directly from a monolithic 
0-1 integer encoding by integer local search, 

— evidence that OIP soft-constraint encodings can capture structural as- 
pects exploitable by integer local search, 

— two problem-specific results: (i) a heuristic can solve a temporally 
dense double-round robin sports scheduling problem, and (ii) a general- 
purpose heuristic can solve a large production planning problem en- 
coded with finite domain variables. 




2. Frameworks for Combinatorial Optimization 



This chapter outlines some successful frameworks for combinatorial optimiza- 
tion. It is meant as a brief introduction into terminology and outlines the basic 
principles of the frameworks which are applied in the subsequent case stud- 
ies. The three frameworks that will be discussed, integer linear programming 
(ILP), finite domain constraint programming (CP) and local search are well 
established, comprise of a variety of techniques, and many successful applica- 
tions have been reported. ILP and CP can be considered the state-of-the-art 
of general-purpose optimization methods, whereas local search should be seen 
as an approach that can be tailored to many different optimization problems 
by adapting its conceptual components to the respective problem context. 
We also discuss successful local search methods for solving propositional sat- 
isfiability problems. 



2.1 Integer Programming Branch-and-Bound 

One of the most successful and widespread approaches to general integer 
programming problems is the IP branch- and-bound algorithm. Here, we will 
discuss the branch-and-bound algorithm that builds on linear programming 
relaxations. It is the basic algorithm used by most commercial codes for solv- 
ing mixed-integer programming problems (MILP). MILP is a superset of pure 
integer programming ILP, in that some of the variables are required to be 
integer, yet others may assume continuous values. The following short expo- 
sition follows Nemhauser & Wolsey [114]; for a more complete introduction 
and for an introduction to the underlying strategy of linear programming 
(LP), the reader is referred to an introductory text in operations research 
[114, 71,31]. 

IP branch-and-bound is concerned with finding optimal solutions to the 
ILP problem, 

(ILP) Zip = min{ cx : x G S' }, where S = {x G Z" : Ax > b }, 

Note that the ILP becomes a MILP problem when some of the integrality 
constraints are relaxed. The branch-and-bound strategy can equally be ap- 
plied to MILP problems. The basic principle underlying branch-and-bound 
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is to divide- and- conquer . The original problem is divided into sub-problems, 
by partitioning the set of feasible solutions into subsets, and recursively solv- 
ing those. This branching strategy continues until the subproblem is either 
small enough to be solved directly or it is known that it can be discarded be- 
cause it cannot possibly contain an optimal solution for the original problem 
{hounding). 

LP relaxation. In IP branch-and-bound, a tree is searched. Initially, at 
the root node, a linear relaxation of the original problem is solved, i.e. S is 
replaced with S'^p = { x G R" : Ax > b }. Subsequently, constraints are 
added to the problem to divide it into subproblems to be solved separately. 
At each node, a linear relaxation is solved, 

2lp = min{ cx : X G S^p }, 

where S’Jp constitutes the linear relaxation of the respective subproblem S'*. 

Also, at any node i during search, there are three conditions when a node 
can be pruned, i. e. none of its children need to be considered. The conditions 
are 

(i) infeasibility, if Sip = 0, i. e. no solution to the linear program exists 

(ii) optimality, if x* G Z”, i.e. the solution is integral 

(iii) value dominance, zip > z^, where z^ is the value of a known feasible 
solution to ILP. 

'Tree Search. If the node is not pruned by one of the conditions (i)-(iii), 
branching occurs. Branching is done by adding linear constraints to the 
current problem. There are various ways to perform branching. One com- 
mon way is to perform ‘variable dichotomy’, that is choose some variable Xj 
which has a non-integral solution value x* in the linear relaxation. Branch 
on S' n {x G Z” : xj < and S n {x G Z” : xj > -I- 1}, 

and recursively search both nodes. At any node i, if the given LP solution is 
optimal, the current feasible solution is stored. 

The size of the enumeration tree depends largely on the pruning, i.e. on 
the quality of the bounds produced by the linear programming relaxation 
and on the solutions found early in the search. If the linear relaxation at the 
root yields solutions whose objective function values are close to the optimal 
solutions of (ILP), we say that the relaxation is tight. 

Many issues need to be considered to develop efficient branch-and-bound 
strategies, such as the selection of branching variables, and more generally 
search strategies to explore the search tree. Efficient branch-and-bound im- 
plementations like Cplex further add valid inequalities (also called cuts) 
which are inferred from specific classes of constraints present in the origi- 
nal constraints. Using these components in addition to efficient algorithms 
for computing and re-computing LP relaxations, ILP branch-and-bound pro- 
vides a general and efficient technique for many ILP optimization problems. 
A variety of efficient commercial branch-and-bound solvers are available on 
the market, (e. g. Cplex, LINDO, XPRESSMP, MINTO). 
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2.2 Finite Domain Constraint Programming 

Finite domain constraint programming (CP) is a programming technique de- 
signed for solving constraint satisfaction problems (CSP). A CSP is typically 
defined [142] in terms of (i) a set of variables, each ranging over a finite 
discrete domain of values, and (ii) a set of constraints, which are relations 
over subsets of the variable domains. The problem is to assign values to all 
variables from their domains, subject to the constraints. A large number of 
problems in AI and other areas of computer science can be viewed as special 
cases of this general notion of CSP [98] . 

Constraint programming evolved from research in constraint logic pro- 
gramming languages ([82] gives a survey), and led to the development of con- 
straint programming languages such as CHIP [42], Oz [137], CLP(FD) [32], 
ECLiPSe [6], and the constraint programming library ILOG Solver [79, 120]. 
Concurrent constraint programming properly addresses the concurrent as- 
pects of constraint programming [129, 70, 137]. CP technology is becoming 
increasingly successful for industrial problems, and many emerging appli- 
cations have been attacked with constraint programming (see [146] for a 
survey). We will next describe the CP framework, following [69, 139]. 

Constraint Store. Every variable of the CSP problem is represented by 
a finite domain variable. All information on the variables is stored directly 
in a constraint store [81], in terms of the sets of possible values that each 
variables can take. This set is called the current domain of the variable. 
Computation starts with the current domain of every variable being the one 
given in the CSP. For example x S {1,...,5} might be the initial current 
domain. Subsequently, the information on the current variable domains is 
updated in the constraint store. This means that the domain information of 
the variables is treated as a primitive constraint. 

Constraint Propagators. In addition to the primitive constraints, com- 
binatorial problems require non-primitive constraints, which are handled 
through computational agents called propagators [153, 138] as illustrated in 
Figure 2.1. Propagators operationalize the semantic definition of the con- 
straints. Some of the corresponding local consistency techniques have been 
established in artificial intelligence [142]. In finite domain constraint pro- 
gramming, the basic inference principle of propagators is domain-reduction. 
Each propagator observes the variables that occur in the constraint. When- 
ever possible, it amplifies the constraint store in terms of these variables by 
excluding values from their domains. 

For example, a propagator can realize a constraint all- different , . . . , x„), 
requiring that all variables Xi, . . . ,x„ be assigned different values. There are 
different ways to operationalize this constraint, some based on sophisticated 
algorithms [127, 121]. A straightforward way is to wait until the domain of 
one of the variables becomes a singleton and then eliminate its element from 
the domains of the other variables. Removing the value from the domains of 
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Figure 2.1. Computational setup of a computation space. 

the other variables may in turn trigger other propagation. If a propagator is 
entailed, it ceases existence. If a propagator can determine that a variable 
domain becomes empty, the constraint store becomes inconsistent. 

The propagation process continues until no propagator can further am- 
plify the constraint store. The computation space (i. e. the constraint store 
together with the propagators) is said to become stable. Still, at this point, 
many variables will typically have multiple values in their domains. Thus the 
constraint store does not represent a solution to the CSP. 

Tree Search. Once a constraint store has become stable, no further infor- 
mation can be inferred by the propagators. To solve the current finite domain 
problem P* at node i, one can choose a constraint C and solve two subprob- 
lems P* U {C} and P® U {^O}. Such branching on constraint C is likely 
to trigger new propagation. After stability is reached again, the branching 
process is continued recursively on both sides. Often, the constraint C is an 
assignment of a variable to one of its remaining values, e. g. x = 1 if {1, . . . , 3} 
are remaining. 

To obtain a complete search for all solutions, the branching continues re- 
cursively until either all variables have a unique value in the store or until the 
store has become inconsistent. The branching scheme yields a complete set 
of solutions to the original problem because of the soundness and complete- 
ness of the branching. The choice of constraint C is a critical issue for the 
performance of the search. If C assigns one variable, an enumeration strategy 
determines which variable and which value from its domain are selected for 
assignment. The issue here is to find good heuristics for variable and value 
selection. A full description of the branching is called a search strategy. 

Finite domain CP systems usually offer rich support for the propagators 
that can be used to model constraint problems. Also CP systems take con- 
trol of propagation and search. Different CP systems offer varying support in 
the design of search strategies. An advanced concept for high-level program- 
ming of search strategies is through using first-class computation spaces [131]. 
By providing the building blocks of constraint propagation, branching and 
search, constraint programming can be viewed as a supporting framework for 
the rapid development of domain-specific optimization strategies. 

CP has seen much success in a variety of domains; for instance in schedul- 
ing, various algorithmic techniques from OR (e.g. [26, 8]) have been inte- 
grated into constraint programming by encapsulation into propagators and 
branching strategies [10, 153]. Thos strategies are available in both academic 
and commercial solvers such as Oz or CHIP, or ILOG Scheduler. 
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2.3 Local Search 

Many combinatorial optimization problems are iVP-hard [50] , and the theory 
of iVP-completeness has reduced hopes that AT’-hard problems can be solved 
within polynomially bounded computation times. Nevertheless, sub-optimal 
solutions are sometimes easy to find. Consequently, there is much interest in 
approximation algorithms (heuristics) that can find near-optimal solutions 
within reasonable running times, even at the cost of giving up the optimality 
guarantee under infinite runtime. 

Practically, for many realistic optimization problems good solutions can 
be found efficiently, and heuristics are typically among the best strategies 
in terms of efficiency and solution quality for problems of realistic size and 
complexity [1, 123, 61, 126]. Heuristics can be classified as either construc- 
tive (greedy) heuristics or as local search heuristics. The former are typi- 
cally efficient one-pass algorithms whereas the latter are strategies of iterative 
improvement. We will be concerned exclusively with local search heuristics 
here. By relaxing the guarantee of finding optimal (or even feasible) solutions, 
heuristics are not required to search for solutions systematically. Instead they 
can freely follow a local gradient in the search space. It is sometimes argued 
that it is this freedom that gives local search strategies an advantage over 
systematic strategies on certain large optimization problems. 

Local search algorithms have been applied to many combinatorial opti- 
mization problems in one or another variation. Local search algorithms can 
be described in terms of several basic components. The combinatorial prob- 
lem to solve, a cost function of a solution to the problem, a neighborhood 
function that defines the possible transitions in the search space. And finally, 
the control strategy according to which the local moves are performed. In 
the following definitions, we will informally follow [1] extended to apply to 
feasibility problems as well. 

Combinatorial Problem. For the purpose of this discussion, a combinatorial 
problem can be specified by a set of problem instances to be solved. A 
problem is either a minimization or maximization problem. 

Cost Function. An instance of a combinatorial problem is defined by the set 
of feasible solutions and a cost function that maps each solution to a 
scalar cost. The problem is to find a globally optimal feasible solution, 
i. e. a feasible solution that optimizes the cost function. We will only con- 
sider minimization problems here. 

Neighborhood Function. Local search progresses by making transitions from 
one node to another. The set of nodes includes the feasible solution, but 
may also include other, infeasible, solutions. Given an instance of a com- 
binatorial problem, the neighborhood function is defined by a mapping 
from the set of nodes to its neighbors, i. e. the subsets of the set of nodes. 
A solution is locally optimal with respect to a neighborhood function if 




16 



2. Frameworks for Combinatorial Optimization 



its cost is strictly better than the cost of each of all its neighbors. Notice 
that there can also be ‘stateful’ neighborhood functions, i.e. the set of 
neighbors can change as the search progresses. 

Control Strategy. The final component is the control strategy. It defines the 
strategy of how the nodes are examined. For example, a basic control 
strategy of local search is iterative improvement. Here, one starts with 
some initial solution and searches its neighborhood for a solution of lower 
cost. If such a solution is found, the current solution is replaced and the 
search continues. Otherwise, the algorithm returns the current solution, 
which is then locally optimal. 

A central problem of local search are local optima, i. e. nodes in the search 
space where no neighbor strictly improves over the current node in terms 
of the cost function. Many strategies have been proposed that address the 
problem of how to overcome local optima. In many cases, non-improving local 
moves are admitted based on a probabilistic decision {noise) or based on the 
history of the search. A number of ‘meta-heuristics’ have been proposed that 
address the problem of local optima. 



2.3.1 Meta-heuristics 

To obtain guiding principles for the design of effective optimization strategies, 
a number of conceptual meta-level strategies have been proposed for local 
search. These strategies are referred to as meta-heuristics , a term coined by 
Glover [56]. Some of the most prominent meta-heuristics for optimization 
simulate aspects of processes that occur in nature^ (like genetic evolution, 
metal annealing or neural processes) and are briefly described in the following. 
Note that these strategies can typically be superimposed on a given basic 
control strategy. 

Simulated Annealing, introduced for optimization by Kirkpatrick et al. in 
1983, uses the metaphor of the annealing process in steal manufacturing 
by which the brittleness of the steal can be reduced [96] . The central idea 
is to accept a candidate move that decreases the solution quality based 
on a probabilistic decision. During the time of the search, the probabil- 
ity of acceptance of such deteriorating moves is decreased according to a 
given annealing schedule [2]. 

Genetic Algorithms model optimization as an evolutionary process. The 
strategy is to have a pool of chromosomes and iteratively apply the prin- 
ciples of mutation, mating and selection to attain ‘survival of the Attest’ 
[74, 62] . Genetic algorithms extend the basic local search scheme to pop- 
ulations of solutions. Early applications in Artificial Intelligence in the 

^ An exception is ‘Tabu Search’ framework. 
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1960s were in game-playing, pattern recognition, or adaptation, and it 
is more recently that applications to optimization problems have been 
reported [62, 125, 113]. 

Tabu Search uses information based on the history of the search [57, 61], 
and is a particularly successful strategy for many practical problems. 
The central idea is the use of adaptive memory to overcome local optima 
by driving the search to different {diversification) or back to promising 
(intensification) parts of the search space. 

Artificial Neural Networks make use of the metaphor of the neuron and are 
organized in network structures. The network of nodes is iteratively mod- 
ified by adjusting the interconnections between neurons according to var- 
ious schemes. Neural networks have been popular since the late 1960s, 
and have more recently been successfully applied to optimization prob- 
lems [78, 44, 119, 38]. 

Meta-heuristics express orthogonal concepts and hybrids are possible. Meta- 
heuristics are domain-independent conceptual strategies. For particular ap- 
plication problems, they require concrete implementation and their success 
varies between different application domains. 



2.3.2 RISC and CISC Local Search 

We can distinguish local search strategies for combinatorial optimization ac- 
cording to the structure of the local neighborhood. We have earlier given a 
short description of Walksat, a local search method for propositional satisfia- 
bility in which the local neighborhood of an assignment A consists of a subset 
oi {A' : A' = A with one variable dipped} and contains 0(n) neighbors for 
a problem instance with n variables. This local neighborhood is simple and 
small. 

On the other hand, many local search strategies use local neighborhoods 
that are more intricate and larger. For example, consider the traveling sales- 
man problem, the problem of finding a minimal-cost tour visiting a number of 
cities such that each city is visited exactly once. Effective local search strate- 
gies for traveling salesman move in the space of feasible tours by complex 
edge-exchanges on tours [89]. Another example is job-shop scheduling, where 
the goal is to assign start-times to a number of tasks that compete for re- 
sources, such that the finishing time of the latest finishing task is minimized 
(makespan optimization) . Effective techniques for job-shop-scheduling move 
in the space of feasible schedules by complex move operations between tasks 
that are on a critical path‘s of the schedule [7]. 



^ A path through the directed graph of adjacent tasks which ends in one of the 
tasks that hnish latest. 
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Such local search strategies differ from the SAT case in that problem 
solutions are expressed in terms of complex entities like tours or schedules . All 
solutions are required to comply with this structure, and to avoid the difficulty 
of restoring it, the local moves are designed to be structure-preserving. This 
is contrasted with local search for SAT and the proposed methods for integer 
local search, in which no explicit solution structure is preserved. 

Here, we exclusively consider strategies which (i) operate by changing 
variable- values, (ii) have neighborhoods with a size linear in the problem 
size (number of variables), and (iii) guarantee to preserve only one property 
with any move, namely that all variables are assigned exactly one value from 
their domain. Because of this atomic nature of the local moves we can refer 
to strategies with properties (i)-(iii) as ‘reduced instruction set’ or RISC 
local search. This is in contrast with the CISC local search strategies for the 
domains traveling salesman or scheduling. We consider both feasibility and 
optimization problem formulations. 

2.3.3 Local Search for SAT 

Local search strategies have recently seen much success for model finding in 
propositional satisfiability [135, 64, 111, 53, 134]. Figure 2.2 gives the outline 
of a typical local search routine in the spirit of the now classic strategy Gsat 
by Selman, Levesque and Mitchel [135]. It searches for a satisfying variable 
assignment for a set of clauses A. Here, local moves are “flips” of variables 
that are chosen by select-variable, usually according to a randomized greedy 
strategy. The measure to perform hill-climbing on for the SAT problem is 
typically the number of satisfied clauses. The parameter Maxtries can be used 
to ensure termination, while Maxfiips determines the frequency of restarts 
that can often help to overcome local minima (in the number of unsatisfied 
clauses). 

Most SAT local search algorithms follow Figure 2.2 and thus have a re- 
markably simple structure on this level of abstraction. The remaining degree 
of freedom in this scheme is the policy to select the next variable to be dipped. 
This policy, in fact, is the key component of SAT local search algorithms and 
is of critical importance for their empirical performance. A variety of policies 
have been described [135, 64, 132, 134, 53, 118, 106] that vary in a static and 
a dynamic aspect. Statically, the probability of a variable to be dipped de- 
pends only on the current variable assignment and on the scheme to compute 
the score of a variable, e. g. how many clauses will be satisfied after the flip 
or how many clauses will break because of the flip. 

Dynamically, the decision can also depend on the history of the search. For 
example, a Tabu element [61] can forbid to undo a recent flip. Alternatively, 
one can break ties between variables with an equal score according to how 
recently a variable has been dipped [53] 

Randomness. The most effective local search procedures for SAT are ran- 
domized algorithms. There are two similar purposes for which randomness 
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proc Local-Search-SAT 

Input clauses C, Maxfiips, and Maxtries 
Output a satisfying total assignment of C, if found 
for i := 1 to Maxtries do 

A := random truth assignment 
for j := 1 to Maxfiips do 

if A satisfies C then return A 
P := select-variable{C, A) 

A'.= A with P flipped 

end 

end 

return “No satisfying assignment found” 

end 

Figure 2.2. A generic local search procedure for SAT. 



is used: (i) to select a starting point for the local search (the initial variable 
assignment), and (ii) to drive the search into different regions of the search 
space (diversification), which can also reduce the dependency of the local 
search on its starting point and thereby make it less dependent on restarts. 

Walksat. While the Gsat algorithm and its descendants choose a variable 
to flip among all the variables that appear in unsatisfied clauses with equal 
probability, the Walksat strategy by Selman, Kautz and Cohen [134, 93] 
favors variables that appear in many unsatisfied clauses: It first randomly 
selects an unsatisfied clause, and then chooses among the variables in that 
clause. Walksat is a particularly effective local search strategy and its variable 
selection policy is given in Figure 2.3. 



proc select-variable(C, A) 

c := a random unsatisfied clause from C 
m := min { b(A, v) : v € c} 

if m = 0 then s := a variable in v € c with b(A, u) = 0 
else with probability p : s := a random variable in c 

probability 1 — p : s := a variable v G c 

with minimal b(A, v) 

end 

return s 

end 

Figure 2.3. The Walksat variable selection strategy, ties are broken at ran- 
dom. 
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Walksat operates as follows. First, it randomly selects one of the clauses 
from c that are not satisfied by the given assignment A. It then selects one of 
the variables in c and flips its value, thereby rendering c satisfied. Because of 
the flip, however, one or more other clauses may become unsatisfied. There- 
fore, to decide which variable to flip, it considers the number of clauses that 
become unsatisfied (break) when one of the variables in c is flipped, for each 
of the variables in cA If it is possible to make c true without breaking some 
other clause, this move is greedily accepted. Otherwise, Walksat follows a 
probabilistic policy. The entire strategy is given in Figure 2.3, where b(A,v) 
denotes the number of clauses in C that break if a variable v is flipped given 
the assignment A. 

Incremental Data Structures. The straightforward way of implementing 
the evaluation of a variable flip (scoring) would be to simulate it. However, 
to efficiently determine which of the many variables leads to the best score, 
Gsat and its variants employ incremental data structures. An example use 
is the number of true literals which is recorded and maintained incrementally 
for every clause. Another is to maintain for every variable the change in the 
number of satisfied clauses when changing the value of the variable from 
false to true, assuming the variable is currently false (and vice versa). For 
a detailed discussion of efficient incremental data structures, see e. g. Michel 
and Van Hentenryck [107]. Incremental data structures allow the algorithms 
to move at a rapid pace in the space of variable assignments. 



2.3.4 Application Domains of SAT Local Search 

Walksat and other flavors of local search for propositional satisfiability have 
been applied to a variety of combinatorial problems. The following list gives 
a short survey on application domains various local search procedures for 
propositional satisfiability have been applied. The domains of study include 
graph coloring, N-queens and Boolean induction [135, 132], circuit diagno- 
sis, circuit synthesis various planning problems (logistics, rocket, towers of 
hanoi, blocks world) [134, 93], Sadeh’s scheduling problems [35, 148], quasi- 
groups, and number factorization [54], and Hamiltonian circuit [77]. Some of 
the above problems have been used in the Beijing SAT competition [101]. 
Further, SAT local search algorithms have been applied to a collection of 
benchmarks within the DIMACS effort on “Cliques, coloring, and satisfia- 
bility” [90], in particular Boolean function synthesis, circuit testing, parity 
function learning, randomly generated one-solution problems [128]. Another 
source of SAT benchmarks which attracted much interest and is random 3- 
SAT [110, 34]. Random 3-SAT has been a driving force for studying and 
improving local search algorithms e.g. [135, 133, 52, 53, 134, 118]. 



® Note that this ‘breakcounf is an approximation of the true change in the number 
of satisfied clauses. 
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Maximum Satisfiability. There are also optimization counterparts of SAT, 
MAXSAT and and weighted and/or partial MAXSAT. The goal in MAXSAT 
is to find an assignment that maximizes the number of satisfied clauses. In 
weighted MAXSAT , a weight is associated with each clause and the sum 
of the weights of the satisfied clauses is to be maximized. Another recent 
variant is partial MAXSAT in which the clauses are classified into hard and 
soft clauses, and the goal is to maximize the number of soft clauses while all 
hard clauses must simultaneously be satisfied. 

Local search methods applied to randomly generated MAXSAT problems 
have been reported [67, 134, 147], partial MAXSAT has been used for en- 
codings of class-scheduling and random formulas [29], and partial weighted 
MAXSAT has been used for solving steiner tree problems [85] . 



2.4 Modeling Languages 

The optimization frameworks given above are computational frameworks for 
solving constraint optimization problems that are given by a suitable set of 
constraints. 

Complementary to the aspect of problem solving, modeling languages have 
been designed to allow to formulate constraint problems conveniently and 
analyze their solutions. An ILP modeling system takes as input a description 
of the constraints and objective functions in some natural formulation. Often, 
this representation is quite similar to the mathematical description. Given the 
model, there is typically some presolving stage performed for simplification 
and feasibility testing [47]. Subsequently, the model is presented to an ILP 
solver that is connected to the modeling system. Finally, after the solutions 
are returned from the solver, the modeling system allows for inspection of the 
solution. Most modeling systems support a variety of algorithmic codes. The 
integer local search procedure that will be given later, Wsat(oip), has been 
hooked up to the AMPL modeling language [49]. Other popular modeling 
systems include GAMS [18], LINGO, PLAM [13]. 

2.5 Search Relaxations and Integer Local Search 

Search strategies for combinatorial optimization can be contrasted according 
to different criteria in order to better understand their characteristics. For 
example, the ‘systematicity’ view contrasts search strategies as to whether 
they are systematic (complete) or follow a local gradient (usually incomplete) . 
For instance, intelligent backtracking and local search can be contrasted in 
this view and combinations have been proposed [55]. 

Another view is the ‘inference’ view, which classifies strategies according 
to which constraints in a system are treated as primitive (i. e. efficient meth- 
ods can be used to solve them) vs. non-primitive (i. e. adding them makes 
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the system hard to solve) [144]. Integer programming and finite domain con- 
straint programming can be contrasted and integrated in this view [21]. 

To improve our understanding of integer local search, let us view combina- 
torial search in terms of relaxations . Combinatorial search means to traverse 
the nodes of a search space, be it in a systematic or non-systematic way, and 
independent of the amount of inference that takes place to get from one node 
to the next. Either way, the essence of combinatorial search is to seek for 
‘perfect’ nodes (in the sense of feasible and optimal solutions) by traversing 
many ‘imperfect’ nodes (solutions that meet only a subset of the requirements 
while violating others). 

If the solutions to a discrete optimization problem are defined by variable 
assignments, the assignments must meet three requirements: (i) totality, i.e. 
all variables have exactly one value assigned, (ii) integrality, i.e. all assigned 
variable values are integral, and (iii) consistency, i.e. the assignment meets 
all problem constraints. We argue here that it is an important factor of a 
strategy which requirements are relaxed during search. 

Integer programming branch-and-bound. The relaxation view of integer pro- 
gramming branch-and-bound strategies is: relax the integrality con- 
straints in order to make use of efficient strategies to solve a linear pro- 
gram. In branch-and-bound, this relaxation occurs at each node of a 
search tree: branching constraints are added, the linear system is solved 
and all variables are assigned values — some of which happen to be inte- 
gral. The search tree is traversed in this fashion until a solution is found 
with all variables integral. Then, the solution is stored, the objective 
function is bounded and the search continues, in the following using the 
relaxation bounds for pruning. Hence, the requirement that is relaxed by 
IP branch-and-bound is integrality, while maintaining that all variables 
be assigned values (totality) and all intermediate LP solutions be feasible 
to the problem constraints (one way to define consistency). 

Constraint satisfaction search. Conversely, complete search strategies for con- 
straint satisfaction (CSP) [142] are based on propagate-and-branch and 
only assign integral values to variables. Here, the variables range over 
finite integer domains and the search starts from a partial assignment of 
the variables. At any point, some variables have a value assigned while 
others are unassigned (more than one value is left in their domain): For 
instance, suppose we start with a variable x € {1, . . . , 5}, which due to 
some constraint a; < 3 becomes x € {1,2}, and next due to branching 
on X might become assigned to a; = 1. The power of CSP methods stems 
from strong propagation algorithms (e. g. arc-consistency) that rule out 
all variable values that are known to be inconsistent with the current 
partial variable assignment and the set of constraints (local consistency). 
Search in CSP progresses in the space of partial variable assignments 
where variable values are speculatively assigned (and possibly later re- 
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tracted) with the goal to eventually assign one value to every variable, 
such that the solution is optimal. Hence, constraint satisfaction search 
maintains integrality and local consistency, but relaxes totality, i. e. the 
requirement that all variables be assigned one value. 

Integer local search. Complementary to the two previous frameworks, integer 
local search relaxes the third property: consistency. That is, all variables 
are always assigned individual values, and all values are integral, yet the 
assignments may violate problem constraints and may thus be inconsis- 
tent . The goal is to find a consistent solution (one which does not violate 
any problem constraints) that is optimal. Thus, integer local search main- 
tains integrality and totality, but relaxes consistency. 

We hope that this view of combinatorial search can foster the understanding 
of integer local search and also stimulate ideas for new hybrid methods. 
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This chapter introduces new local search strategies for integer optimization 
and represents the technical core of this book. It describes and discusses 
Wsat(oip), a domain-independent method that generalizes the Walksat pro- 
cedure of Selman, Kautz, and Cohen [134] for propositional satisfiability to 
integer optimization and integer constraint solving. For performance on real- 
istic applications, the method additionally incorporates principles from Tabu 
Search [58]. 

Wsat(oip) operates on over-constrained integer programs (OIPs), an al- 
gebraic representation for combinatorial optimization problems which is sim- 
ilar to integer programs. The chapter first introduces and discusses OIP, 
showing that OIP is a special case of integer linear programs. We will argue 
that OIPs are well suited for devising efficient iterative repair strategies for 
integer optimization. On the other hand, OIPs allow for combinations with 
linear programming via a reduction to integer linear programs. 

Then the Wsat(oip) procedure will be described in terms of its basic 
principles and we give the details of a carefully engineered strategy for the 
selection of local moves that evolved in the course of the case studies. 

Several combinations with linear programming will then be discussed for 
computing bounds on the optimal solution, approximation and search space 
reduction. Finally, we will illustrate the basic algorithms and several exten- 
sions by a graphical example to review the described techniques. The chapter 
concludes by discussing related work on local search for integer programming 
and constraint satisfaction. 



3.1 Over-Constrained Integer Programs 

The first step in solving an optimization problem is to choose a suitable repre- 
sentation. In Artificial Intelligence, a popular representation is propositional 
satisfiability (SAT). SAT can represent a variety of interesting combinato- 
rial problems, for example graph coloring [135, 132], circuit diagnosis and 
synthesis [91, 134], or various AI planning problems [93, 94]. 

A number of efficient search strategies have been developed for SAT in 
recent years, both complete [34, 40] and incomplete [135, 134, 64] (the corre- 
sponding OR terminology is exact vs. heuristic). However, many combinato- 
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rial problems have no concise encoding in propositional logic, especially those 
involving arithmetic constraints. Hence these algorithms cannot be applied. 

For example, consider the pigeonhole problem which occurs as a core 
problem within many combinatorial problems. Consider a statement with 
Boolean variables pij, where pij means pigeon i is in hole j . A natural encod- 
ing is to use two different constraints, (a) every pigeon is in exactly one hole 

Pij = 1 (for all i), and (b) no two pigeons are in the same hole ^^Pij < 1 
(for all j). Given n pigeons and m holes, this formulation consists of n -I- m 
pseudo-Boolean constraints. On the other hand, a SAT encoding would be (a) 
Vj Pij (for all i) and VjVfc yf j : pij — > piF (for all i, 0{m?n) clauses); simi- 
larly for (b). With 0{rn?n + n^m) constraints, the size of the SAT encoding 
would be impractical for larger instances. 

Pseudo-Boolean / 0-1 Integer Constraints. The class of linear pseudo- 
Boolean constraints (linear 0-1 integer constraints) is defined as follows [66]. 
A linear pseudo-Boolean constraint is of the form 

• L, ~ (i, (3.1) 

iGl 

where Ci, d are rational numbers, {=)<)<)>)>}) and the Li are literals 
for alH G / (a literal is a Boolean variable or its negation). A set of pseudo- 
Boolean constraints together with an objective function yields a 0-1 ILP. 
Pseudo-Boolean constraints generalize SAT in the sense that every Boolean 
clause (disjunction of literals) can be translated into a single linear pseudo- 
Boolean inequality. For example, the clause x \/ y would be translated to 
(1 - a;) -h p > 1. 

On the other hand, when converting linear pseudo-Boolean inequalities to 
propositional satisfiability, the number of SAT clauses required to represent 
one inequality can grow exponentially with the number of variables in the 
inequality. To see this, consider an example from Barth [12], the conversion 
of /i -I- • • • -I- > d. Without introducing new variables, its equivalent SAT 

representation is a conjunction of SAT clauses, 

f\ i'diei h)- 

I<Z{l,...,n}-.\I\=n-d+l 

Generalizing SAT Local Search. There are still practical shortcomings of 
the pure pseudo-Boolean constraint representation (3.1). First, optimization 
problems (as opposed to decision problems) contain objectives which should 
be included in the representation. 

Second, certain problems may better be represented using non-Boolean 
decision variables (e. g. production of a good could vary between 1 and 100 
items). Compiling such problems to a pseudo-Boolean system would again 
incur an increase in both the number of variables and the length of the 
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constraints.^ Depending on the applied algorithms, the structure present in 
the original formulation may not be accessible in the compiled version of the 
problem. 

Despite the fact that SAT does not quite have the right expressivity for 
many realistic optimization problems, efficient methods exist for SAT and 
it is desirable to generalize their principles to more expressive constraint 
classes. Obtaining leverage from a generalization is especially interesting for 
the efficient SAT local search algorithms developed recently, e. g. Walksat 
[134]. 

Over-constrained Integer Programs: Motivation. To give a natural 
generalization, we will introduce over-constrained integer programs (OIPs), 
an algebraic representation that is suited for a range of combinatorial opti- 
mization problems. With respect to expressivity, we will show that OIPs are 
a special case of integer linear programs (ILPs). 

The principal difference to ILPs is that while ILPs use a monolithic ob- 
jective function, OIPs represent the overall optimization objective by many 
competing soft-constraints. Using soft constraints to encode objectives, it is 
natural to apply iterative repair [109] to integer optimization. Further, the 
OIP structure can be exploited by iterative local search: After a short initial 
phase, only a small fraction of the soft constraints are violated and the search 
can focus on repairing those violated constraints. 

In general, an over-constrained system is a set of constraints in which 
typically not all constraints may be satisfied and some are therefore marked 
as ‘soft’ or ranked according to a preference system [84]. Optimization objec- 
tives have also been encoded by soft constraints in the context of constraint 
hierarchies [22]. 

OIPs are similar to ILPs. So can we make use of techniques from linear 
programming? The answer is yes. We will give a transformation of OIP into 
ILP that is inspired from piecewise-linear programming. Another aspect con- 
cerns standard modeling languages like AMPL [48]. Can they be applied to 
model OIPs? The answer is, again, yes. ILP modeling languages can be used 
for modeling. In fact, combining a modeling language with a local search al- 
gorithm directly yields a practical constraint solver. For illustration, we will 
provide a detailed AMPL model of one case study from sports scheduling in 
the Appendix. 



3.1.1 Definition 

We refer to a constraint system of hard and soft inequalities over finite do- 
main variables as an over-constrained integer program, {OIP). Here, we only 
consider the case where all constraints are linear inequalities and the system 
is given by a tuple O = {A, b, C, d, D), formulated in matrix notation as 

^ Binary and gray code representation for large variable domains are possible [105], 
but we will see some limitations for local search in Chapter 7. 
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Ax > b 

Cx < d (soft) (3.2) 

X G D, 

where A = (a^) and C = (cy) are m x n coefficient matrices, b and d are 
m- vectors, and x = (xi, . . . , Xn) is the variable vector, ranging over posi- 
tive finite domains D = (Di, . . . , (3-2) will be interpreted as the OIP 

minimization problem 

min{ ||Cx — d|| : Ax > b, x G D}, with |jv|| := max(0, Vj). (OIP) 

i 

In II . II , the contribution of each violated soft constraint to the overall objective 
is its degree of violation. An assignment to all the variables that satisfies all 
hard constraints of O is called a feasible solution, and for every feasible 
solution s, the value of HCs — d|| will be called soft constraint violation of O. 
As will be shown, OIPs are a special case of integer linear programs. 

Based on OIPs, a single strategy will later be formulated both to find 
feasible solutions to difficult problems of hard constraints, and to find good 
solutions to optimization problems. The strategy will proceed by iteratively 
repairing violated hard and soft constraints. 

3.1.2 Relation to Integer Linear Programs 

For many practical applications, we would like to enable collaboration of OIP 
solvers with existing optimization technology based on linear programming. 
Examples of such combinations are lower bounding techniques (like linear 
relaxations or Lagrangean relaxation), approximation algorithms based on 
linear programming, or search space reduction techniques (see Section 3.3 for 
a detailed discussion). 

Due to the evaluation ||.||, however, a given OIP is not immediately equiv- 
alent to an integer linear program (ILP). Next, it will be shown that every 
OIP can be converted to an ILP. We consider the following cases: (i) If a 
given OIP has a certain property (called confinedness), it is equivalent to an 
ILP. Further, (ii) every OIP can be converted to an equivalent ILP. 

Equivalence is defined here in the sense that two ILPs/OIPs are equivalent 
if they have the same feasible solutions and their objective function values 
are the same for every feasible solution. 



^ For conciseness, we will discuss OIPs in the form (3.2) which could be referred 
to as min-normal form. Every OIP minimization problem can be converted into 
min normal form by multiplying every incorrectly directed inequality (e. g. < 
instead of >) by — 1 and converting every equality into two inequalities. Input 
to the algorithms described in the following is not required to be in min-normal 
form. Also, implementations use sparse matrix representations. 
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Notational Convention. To simplify the following discussion, we will write 
minjcx : Ax > b, x S D}, 
as an abbreviation for the ILP 

minjcx' : Ax' > b, a;' = ^ j • j/y , ^ yij = 1, G {0, 1}}, (3.3) 

je£>i ieOi 

in which the finite domain variables xi G Di are replaced by variables x[ and 
additional binary variables j/y G {0,1} are introduced for all j G Di. If all 
finite domains range over integers (no loss of generality) then (3.3) is an ILP 
(xGZ'}). 

Definition 3.1.1 (Confinedness). An over-constrained integer program 
(A,b,C,d,D) is confined if and only if for every feasible solution s, the 
following holds: 

Cs-d> 

Proposition 3.1.1. A confined OIP (A,b, C, d,D) is equivalent to the ILP 
min{ ||Cx — d||i : Ax > b, a: G D|, with \\-v\\i := (3.4) 

i 

Proof. As confinedness holds, ||.|| is equivalent to the norm ||.||i for all feasible 
solutions. □ 

The following proposition shows that every OIP can be converted to an 
ILP by introducing additional constraints. 

Proposition 3.1.2. An over-constrained integer program (A, b, (c^), d, D), 
Ax > b, CiX < di (soft ), X G D 
is equivalent to the ILP 

min|y^ Cj : Ax > b, CiX-\- Si — Ci = di, Ci,Si>Q, x G D|. (3.5) 

i 

Proof. The basic idea behind the conversion is that every soft inequality can 
be converted into an equality constraint with two additional variables: a slack 
variable Si and an excess variable e^, where the excess variable accounts for 
the incurred penalty. The OIP minimization problem is 

min{^^ max(0, Ci X — di) : Ax > b, x G D|. 
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To show equivalence, it suffices to show equality of the two objective functions 
for all X. We will employ (componentwise) ordered tuples (a, 6). It remains 
to show that for all x, 

min{(x,eiH h e^) : c* x + Sj - Cj = d*, Si>0,ei>0} 

= (x, max(0, Cl X — di) + • • • + max(0, x — dm))- 

This can be shown componentwise for all i because the Si, Ci are independent 
from Sj,ej for all i j- For each i, we consider the components (x, e*) = 
min{(x, Ci) : . . . } and (x, max(0, c^x — di)). We consider two cases: 

1. Cl X — di < 0. The right-hand-side is max(0, x — di) = 0. The left-hand- 
side also yields e* = 0 since this is the smallest > 0 satisfying d = 

Cj X Si di- 

2. CiX — di > 0. Now the left-hand-side yields 

e* = min{ei : e* = c* x -|- Si - di, Si > 0, ei > 0} = c* x - di, 
which is also the value of the right-hand-side. q 

In summary, OIPs are a special case of ILPs. We notice that all OIP 
models which will be given in the case studies happen to be confined, which 
is typically easy to check. ^ The algorithms that will be described subsequently 
are directly applicable to general OIPs, however. 

There is a close relation of OIP to integer linear programs with piecewise- 
linear objective functions.^ A piecewise-linear function is a function that is 
pieced together from linear segments. Every soft constraint cx < d gives 
rise to a penalty that is expressed by a piecewise-linear convex function 
max(0,cx — d), as shown in Figure 3.1. Given the fact that the sum of two 
convex functions is a convex function, OIP objectives are piecewise-linear 
convex. Piecewise-linearities are employed in models to give a more realistic 
description of costs than can be achieved by linear terms alone [48]. There 
are also extensions of the simplex algorithm for piecewise-linear convex pro- 
gramming [46]. 

An OIP Example. As an example of a piecewise-linear objective function, 
consider the problem of assigning a set of tasks T to a workforce W. Suppose 
that every task consumes one hour and every worker w is employed for 8 
hours, but can work overtime at some cost. The problem can be modeled with 
binary variables, atm = 1 iff task t is assigned to worker w. The constraints 
are: 

® Note that the following is a simple sufficient condition for confinedness: for every 
soft constraint cx < d, there exists a hard constraint cx > d. 

^ I thank Andrew Parkes for bringing my attention to the relation between soft 
constraints and piecewise-linear functions, which eventually lead to Proposition 

3.1.2. 
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(a) cx < d (soft) (b) cx > d {soft) 

Figure 3.1. Piecewise-linear penalty functions 



Assign every task 
Limit workday 
Minimize overtime 



Y.W at™ = 1, for all t. 

at™ < 12, for all w. 

X)t at™ < 8 (soft), for all w. 



The goal is to assign the tasks to the workers such that the summed over- 
time is minimized. The problem as stated above is not a hard combinatorial 
problem but it may occur as subpart of one. 



3.1.3 Constraint-Bounds 

A soft constraint cx < d of an over-constrained integer program is similar 
to an objective function with a bound. Therefore, the right-hand side d will 
be referred to as constraint-bound. For modeling, we need to understand the 
degrees of freedom of choosing the constraint bounds: Consider an OIP O 
containing a soft constraint s : cx < Z, to minimize the excess of a function 
cx over 1. We observe that the larger the values of I, the larger the (soft) 
feasible region of s. Sometimes, for a given value of I there may be no feasible 
solution to the problem that satisfies s. This raises the question in what 
range the constraint-bounds can be increased or decreased without changing 
the optimization problem? 

Definition 3.1.2 (Rebounding). Given an OIP O = (A, b, C, d, D), the 

OIP (A,b,C, d',D) is a called a rebounding of O. 

Proposition 3.1.3 (Invariance under Confined Rebounding). If Oi 

and O 2 are confined OIPs, and O 2 is a rebounding of 0\, then the set of 
optimal solutions is the same for Oi and O 2 ■ 

Proof. Since Oi is confined, it is equivalent to an IP with minimization 
objective ||Cx — di||i. As O 2 is also confined and a syntactic rebounding of 
Oi , its corresponding IP has the same set of constraints but a minimization 
objective of ||Cx — d 2 ||i = ||Cx — di||i -I- fc, for some k. Obviously, every 
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solution X of Oi with objective value s is a solution of O 2 with objective 
s + k and thus the sets of optimal solutions are the same. □ 

Proposition 3.1.3 implies that if for operational purposes it is helpful to 
increase the constraint-bounds, this may be done provided that confinedness 
is maintained. 

Proposition 3.1.4. Let (^,b, C, d,D) be a eonfined OIP. For every d' with 
d'i < di for all i, the rebounding {A,h,C, d',D) is eonfined. 

The proof follows directly from the definition of confinedness. Together 
with Proposition 3.1.3, Proposition 3.1.4 states that optimal solutions do not 
change if the soft constraint-bounds are tightened (note that their objec- 
tive function values do, however). This tells us that choosing soft constraint- 
bounds too tightly does not affect the optimization problem except for a shift 
in the objective function values. 



3.2 Integer Local Search: Wsat(oip) 

This section introduces Wsat(oip), a local search method to solve optimiza- 
tion and feasibility problems represented by OIPs. Starting from some initial 
assignment, the procedure performs changes of variable values, thereby mov- 
ing in the space of integer assignments to find feasible or good solutions. 

Wsat(oip) generalizes Walksat (described in section 2.3), but its per- 
formance on realistic problems is often critically dependent on the incor- 
porated concepts from Tabu Search [61]. As the case studies will demon- 
strate, despite its conceptual simplicity, Wsat(oip) is surprisingly effective 
in terms of performance and robustness. Historically, Wsat(oip) builds upon 
Wsat(7^,8) [149] which performs stochastic tabu search on over-constrained 
pseudo-Boolean systems. Generalizing from Boolean variables to general fi- 
nite domain variables, Wsat(oip) subsumes Wsat(7^,8). 

The Strategy. We describe the method starting from its basic principles, 
and proceed by giving the main loop of the algorithm and the (important) 
details of how the local moves are selected. A summary of the parameters 
concludes the description. Section 3.4 will further illustrate the search process 
and several extensions by a graphical example. 

Wsat(oip) follows an ‘iterative repair’ strategy and operates on a total 
assignment (an assignment to all variables). Individual variable/ value pairs 
are iteratively selected to be changed in order to improve the local gradient 
of an overall measure of the satisfaction of the constraints. The main cycle 
for selecting local moves is illustrated in Figure 3.2. Generalizing from the 
Walksat algorithm [134], variable changes in Wsat(oip) are selected in a two- 
stage strategy of first randomly selecting an unsatisfied constraint for partial 
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two-stage move selection 




restart 



Figure 3.2. Local search and the two-stage control strategy of Walksat. 



repair and from the constraint selecting a variable to be changed.® This two- 
stage control strategy, which we will call ‘ Walksat-Principle', distinguishes 
Walksat among the many flavors of recent local search algorithms for SAT and 
CSP (including Gsat [135], GSAT-|-walk [134] and Min-Gonflicts [109]).® 
Note that it favors those variables that appear in many unsatisfied constraints 

[134]. 

The criterion for move selection is to perform hill-climbing on a score 
which reflects both the degree of infeasibility and the optimization objective. 
A value change of a Boolean variable is to flip (complement) the variable 
to improve the score. As a generalization, a move of Wsat(oip) consists of 
triggering the value of a finite domain variable to a smaller or greater value 
in the neighborhood of its current value assignment. Occasionally, a restart 
with a new initial assignment takes place to escape from local optima, for 
example after a fixed number of moves. 

3.2.1 The Score 

To describe the move selection strategy for over-constrained IPs in detail, we 
first need a score definition. Given a particular assignment x, a system of the 
form (3.2) is evaluated as, 

score(x) = jjb — Axj|> -|- jjCx — dj|. (3.6) 

using the usual norm j|vjj = ^jmax(0,t>i). A useful property of the score 
(3.6) is that it is identical to the value of the objective function of the equiv- 
alent ILP. 

Additionally, the score (3.6) uses a vector A > 0 for weighting the hard 
constraint violations, defined by l|vj|;^ := Aimax(0,Ui). Note that the soft 
constraints do not carry weights in order that the score of feasible solutions 
be equal to the objective function value. Only one case study will make use 
of non-unit weights, and all experiments were performed with weights that 
were assigned statically. 

® In contrast to SAT, a variable change does not always repair the selected con- 
straint completely. 

® More generally, this two-stage control strategy is (i) select a constraint c for 
repair, and (ii) select a partial repair for constraint c. 
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• = feasible integer point 




Figure 3.3. Manhattan distance, a shortest path to enter the feasible region 
of a constraint with all coefficients from {—1, 0, 1}. 



Graphical Illustration. To illustrate the score (3.6), assume a current variable 
assignment s and a constraint c : cx> d with all elements of c from { — 1, 0, 1}. 
Then, ||fi— cs|| measures the number of unit variable changes required to enter 
the feasible region of the constraint, i. e. the Manhattan distance of s to c. 
Figure 3.3 illustrates the situation for the constraint x\ + >6 over two 

variables si, S 2 € {1,2,..., 5}. The score of s = (1, 2) is 6 — (1 + 2) = 3 unit 
variable changes. Section 8.2 will discuss a refinement of this scoring scheme. 

3.2.2 The Main Loop 

In each iteration, Wsat(oip) makes a change of exactly one variable-value 
pair. Figure 3.4 illustrates the basic main loop of Wsat(oip). If an assign- 
ment is found that is better than the best one found in the past, this new best 
assignment is stored. If an assignment is found that is known to be optimal 
(e. g. using a relaxation-proof), the optimal assignment is returned. A restart 
is performed after a fixed number of iterations. 



3.2.3 Move Selection and Tabu Search Extensions 

The fundamental principle behind Wsat(oip) is steepest-descent (selecting 
local moves that most improve the total score) combined with adaptive mem- 
ory [58] and a noise strategy to overcome local minima. The remaining degrees 
of freedom are how to select (i) a constraint and (ii) a partial repair, i.e. a 
variable and its new value. It a time-consuming engineering task to find good 
strategies for (i) and (ii) which has a strong impact on performance. The 
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proc Wsat(oip) 

input OIP O, Maxmoves, Maxtries 
output an approximately optimal feasible solution 
for O, if found 
for f := 1 to Maxtries do 

a := initial total assignment, a[u] G dom{v), 
possibly infeasible 
u := oo 

for j := 1 to Maxmoves do 

if a is known to be optimal then return a 
if a is feasible A score{a) < u then u := a 
c := select-unsatisSed-constraint(0, a) 

{v, s') := select-partial-repair{0 , c, a) 
a := a[u <— s'] 

end 

end 

if u < oo then return u 

else return “no feasible solution found” 

end 

Figure 3.4. Main loop of Wsat(oip) for over-constrained integer 
programs. 



influence of different variable selection strategies on performance has been 
investigated previously for SAT local search [118, 106]. Throughout the case 
studies, unsatisfied constraints are selected at random. Although different 
constraint selection schemes have been studied (among those selecting one of 
the violated constraints ordered by increasing/decreasing ‘constrainedness’), 
we could not find a selection scheme that improved over random selection. 

Figure 3.5 gives a scheme that combines successful elements from stochas- 
tic local search and deterministic tabu search. The particular strategy evolved 
in the course of our case studies and includes a tabu mechanism, history- 
based tie-breaking, and timid noise: A tabu mechanism with tenure of size t 
avoids assigning a variable- value pair that has been assigned in the previous 
t moves — unless the score would improve over the best past score (in tabu 
search terminology, overriding the tabu-status of a move is called aspiration). 
All ties between otherwise equivalent variable- value pairs are broken by a his- 
tory mechanism inspired by Gent and Walsh’s Hsat [53] : On ties, choose the 
move that was chosen i) least frequently, and then ii) longest ago. 

In contrast to SAT local search, the length of 0-1 inequalities that can 
be handled efficiently can be quite large. This is because only those variables 
need to be scored that contribute to the violation of a constraint. For example, 
in the course assignment problems, inequalities contained up to 350 variables 
of which typically only 10% needed to be scored for repair. 
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1. Randomly select an unsatisfied constraint a (with probability 
Phard ^ hard constraint, and with 1 — Phard ^ soft constraint). 

2. From a, select all variables which can be changed such that 
Qf’s score improves. For each such variable, select one or more 
a-improving values and compute the hypothetical total scores 
(Boolean variables are dipped, finite domain variables are trig- 
gered up or down by at most d stpes). 

3. From the selected variable-value pairs, remove the ones which 
are tabu (tabu-aspiration by score). 

4. Of the remaining variable- value pairs, select one that most im- 
proves the total score, if assigned. Break ties according to i) fre- 
quency and ii) recency. 

5. If the total score cannot be improved: With probability Pnoisej 
select a random a-improving non-tabu variable-value pair. With 
1 — Pnoisej Select the best possible one. 

Figure 3.5. A stochastic tabu search strategy for move selection in 
Wsat(oip). The strategy extends Walksat to systems of hard and soft con- 
straints (1.) and its variable selection to general finite domain variables (2.). 
Additionally, the straegy is merged with adaptive memory from tabu search 
(3.) and an extended version of history-based tie-breaking (4.) [53]. 



Parameters. Finally, Table 3.1 summarizes the parameters of the basic al- 
gorithm. It also reports parameter ranges as rules of thumb. Most parameter 
settings in the case studies were performed with settings from these ranges 
(the exact values will be reported) . Additionally, for practical purposes a seed 
parameter is used: fixing the seed yields a deterministic algorithm. 



Table 3.1. Parameters of Wsat(oip) with standard ranges. 



parameter 


std range 


description 


Pnoise 


0-0.2 


probability of a allowing random downhill 
move 


Phard 


0.8-0.9 


probability of selecting a hard constraints for 
repair 


Pzero 


0.5-0.9 


probability of initializing a variable with zero 


t 


1-2 


tabu-tenure (overridden by score-aspiration) 


Maxstep 


2 


maximal trigger distance of non-binary vari- 
ables 


tie-breaking 

Maxtries 

Maxmoves 


on 


history-based tie breaking (frequency,recency) 
total number of tries 
number of moves within a try 
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3.3 Combinations with Linear Programming 

Local search methods like Wsat(oip) suffer from several principal drawbacks 
when used in an optimization context. These are: 

(i) The inability to detect if a (specific) obtained solution is optimal or to 
estimate how far from optimality the solution is. 

(ii) The lack of any guarantee of the quality of the solutions returned. (For 
some NP-hard problems, approximation algorithms exist that can give 
such, although weak, guarantees [73]). 

(iii) The absence of sound search-space pruning methods during the local 
search process. 

We will argue that combinations with linear programming can help to 
overcome these drawbacks to some extent. Especially, for (i) lower bounding, 
(ii) initial assignment by rounding, and (iii) search space reduction. 

The first section describes the standard use of the linear relaxation to 
obtain (lower) bounds for (minimization) problems. The second section de- 
scribes the computation of initial assignments by linear programming followed 
by randomized rounding, and a novel extension of this idea for Wsat(oip). 
And the third section discusses the use of ‘reduced costs’ for search space 
reduction, as proposed by Balas and Martin [9] . 



3.3.1 Bounds from LP Relaxations 

It is well-known that linear programming problems are polynomially solvable 
in theory, and can often be solved efficiently in practice [31]. This fact can be 
exploited by using linear programming (LP) to efficiently compute bounds 
on the optimum of a problem at hand. In practice, bounds obtained directly 
from the LP relaxation are often valuable estimates of the solution quality. 

Consider the following 0-1 integer problem in matrix notation (note that 
LP relaxations are applicable to general integer programs) 

Z\p = minimize cx 

subject to Ax > b (3.7) 

Xi G {0, 1}. 

One way to generate a lower bound to problem (3.7) is to relax the integrality 
constraints xi G {0,1} by substitution with bounds on the variables. This 
yields the linear program 

Zlp = minimize cx 

subject to Ax > b 

Xi G [0, 1]. 



(3.8) 
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As (3.8) is a relaxation of (3.7), every lower bound of (3.8), in particular its 
optimal solution, is a valid lower bound of (3.7). In absence of the optimal 
objective value for the integer programming problem, this fact can be used 
to estimate the quality of a solution to (3.7) by solving the linear relaxation 
(3.8). 

The difference between the optimal IP and LP solutions, Z\p — Zlp, is usu- 
ally referred to as integrality gap. Practitioners have observed that problems 
with small integrality gaps tend to be solved more efficiently by integer pro- 
gramming branch-and-bound than problems with large gaps. Other general 
methods for lower bounding could be used as well, e. g. Lagrangean relaxation 
[17]. 

3.3.2 Initialization by Rounding LP Solutions 

Solving the linear program followed by rounding the non-integral solutions 
can be combined with integer local search in useful ways. Consider a combi- 
natorial optimization problem given by the ILP 

minimize cx 

subject to Ax > b (3.9) 

Xi integer 

As before, the first step is to relax the integrality constraints and apply linear 
programming to solve the system 

minimize cx 

. (3-10) 

subject to Ax > b. 

The result is an LP-optimal solution Xi,i = 1 . . .n. As the Xi may be frac- 
tional, the LP solution may not constitute a feasible solution for the integer 
program. In order to restore integrality, the resulting values Xi can there- 
fore be rounded (i) deterministically or (ii) in a randomized fashion (up or 
down) to yield values Xi. There are several benefits that can be obtained 
from rounding, and we will start by describing two practical benefits that 
arise irrespectively of the employed rounding scheme. Subsequently, we will 
touch on a theoretical benefit of the combination. 

Practical Benefits. Some discrete optimization problems exhibit a low level 
of discreteness because even although all variables are integer, their domains 
are large. An efficient way to solve such problems is by linear programming 
and rounding in the initialization stage of an integer local search method. 
To illustrate this, consider the following problem formulation, occurring as a 
subtask of a real configuration problem.^ 



^ Thanks to Thomas Axling for providing the example. 
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Zip = minimize x\ + 20 x 2 + 20 x 3 + 20 x 4 
subject to xi + 4x2 + 2 x 3 + a ^4 > 800 
a^i + a;2 + 4x3 + xa > 100 
Xi + X2 + X3 + 4X4 > 600 
Xi + X2 + X3 + X4 > 400 
Xi e {1,...,1000}. 

Due to the large variable domains {1, . . . , 1000}, this problem is in a sense 
‘close’ to its linear relaxation. This fact is illustrated by Table 3.2 which 
contains the optimal LP and IP solutions. Using the optimal LP solution and 

Table 3.2. Optimal solutions for a subtask of a configuration problem (3.11). 



var 


LP 


IP 


Xi 


200 


201 


X2 


133.33 


133 


X3 


0 


0 


X4 


66.667 


67 


z 


6000 


6010 



deterministically rounding it yields a solution which violates only the first 
constraint in problem (3.11). 

Because of the type of local moves, Wsat(oip) as described above is not 
the method of choice for problems with very large domains if started from 
a random initial variable assignment. In fact, in this particular example, 
it behaves poorly. However, when initialized with the rounded LP solution 
from Table 3.2 it only takes one variable flip to restore feasibility and arrive 
at the optimal IP solution. This is because examining the ways to repair the 
only violated constraint, all variables of the first constraint are scored and 
increasing xi yields the smallest increase in the overall score. 

Observations for Binary Variables. Even if all variable domains are bi- 
nary, rounding the non-integral solutions obtained from linear programming 
can be helpful. In general, feasibility with the constraints is lost due to the 
rounding step. However, integer local search can be expected to recover fea- 
sibility quickly in most cases. An additional benefit may arise with problems 
for which the LP relaxation yields a large number of integral values. For in- 
stance on “large-sized” generalized assignment benchmarks from OR-library 
[16], instances exist for which the LP optimum has only 1% (5 out of 500!) 
non-integral variables; all other variables were 0 or 1. 

Leashed Local Search. For the case that a significant number of integral values 
are obtained from the LP relaxation, we propose an extension of Wsat(oip) 
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which we call leashed local search. Starting from an initial rounded LP so- 
lution, perform local search as usual. However, limit the radius the search 
may divert from the initial assignment, by limiting the hamming distance 
between the initial solution and the current solution (the hamming distance 
between two Boolean variable assignments is the number of bits that differ). 
We did not explore this technique in the case studies because the principal 
aim of this work is to examine what can be achieved by local search; the 
effectiveness of Leashed Local Search on the other hand would be tied to the 
question how close to the IP solution a corresponding LP solution is. Also, 
in some of our case-studies (see Section 6.1), solving the LP relaxation took 
orders of magnitude longer than solving the original integer problem with 
Wsat(oip). 

Approximation by Randomized Rounding. The approach of random- 
ized rounding is due to Raghavan and Thompson [122], and can be used to 
formulate efficient (polynomial) approximation algorithms with provable per- 
formance guarantees. The key insight of randomized rounding is that certain 
performance guarantees can be derived if one solves a linear relaxation (3.11) 
and randomly rounds the fractional variables. Generally, if xi is the solution 
obtained from linear programming, we assign a solution variable for the IP, 
Xi = 1 with probability Xi and Xi = 0 with probability l—Xi. Recently, a num- 
ber of approximation algorithms for AP-hard problems have been presented 
based on randomized rounding, whose characteristic is to give a quality guar- 
antee for the returned solution [73]. One example is maximum satisfiability 
(MAXSAT) for which randomized rounding can be combined with random 
variable assignment {pzero = 0.5) to yield an approximation algorithm which 
guarantees that its solution satisfies at least 3/4 of the clauses [154, 112]. 

If, for a particular problem, such a performance guarantee exists for ran- 
domized rounding, this guarantee is of course directly inherited by an integer 
local search method if randomized rounding is used to obtain the initial so- 
lution. 



3.3.3 Search Space Reduction Using LP Reduced Costs 

We next describe a method for dynamic search space reduction of local search 
which can be employed in combination with Wsat(oip), and which has been 
reported by Balas and Martin [9] and Abramson et al. [4]. 

The idea is that solving the LP relaxation of an integer program to opti- 
mality reveals information about the sensitivity of the solution with respect 
to changes in the problem’s parameters. In mathematical programming, such 
analysis is referred to as sensitivity analysis [31, 152]. 

An important instrument in sensitivity analysis are reduced costs: When 
solving the linear program to optimality using the simplex method, together 
with the optimal solution one obtains an optimal tableau which contains 
basic variables (the variables that are non-zero in the optimal solution) and 
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nonbasic variables (the variables that are zero) . We describe the idea in terms 
of 0-1 integer programs. 

For any nonbasic variable Vi, its reduce cost is the amount by which 
Vi’s objective function coefficient must be improved before Vi will become a 
basic variable in some optimal solution to the LP. Another interpretation of 
Vi is that it is the amount by which the objective function Z\_p will increase 
(in a minimization problem) if the variable is increased by one. 

Reduced costs can be employed for a dynamic pruning of the search space 
in the following way. If at any time during the search the best feasible solution 
found (upper bound) has an objective function value of then all variables 
Vi can be fixed to zero for which Zip+Vi > Z[jb- This is true because assigning 
one to these variables would yield a solution which is provably worse than 
the best solution already found. 

This strategy can be used to extend Wsat(oip) by allowing it to fix 
variables in the following way: Every time an improved feasible solution (up- 
per bound) with objective value Z\jb is found, all variables Vi for which 
■^LP + Ti > Z[jB holds can be fixed to 0. Notice that the current value of 
Vi must then be 0 (because if Ui = 1 then it would follow that Z\jb > .^lp + ?'i 
which is in contradiction with Z\jb < Zpp + Vi). If all variables are fixed, the 
upper bound is provably optimal. 

A requirement for the pruning to be effective is that a significant per- 
centage of the reduced cost values be sufficiently large. For example, this 
is typically the case for set-partitioning problems where many variables can 
be fixed by the technique.® In the case studies conducted here, except for 
Chapter 7, the reduced costs were not significant (the particular covering 
and assignment problems had mostly zero reduced costs and the feasibility 
problems have no true reduced costs because there is no objective function). 

Preliminary Results. Because of the small reduced costs in our case stud- 
ies, we have not evaluated search space reduction in detail at this point. 
Nevertheless, in addition to the observations by Abramson et al. [4], our ob- 
servations on generalized assignment benchmarks from OR-library [16] are 
very promising. For example, on a ‘large-sized’ GAP instance, 334 out of 500 
binary variables would have been fixed to 0 during the search process us- 
ing the upper bound found by Wsat(oip) after a short time — a remarkable 
reduction of the search space. 

3.3.4 Implementation Issues 

Wsat(oip) has been implemented in C/C-l— k, making use of the Lex/Yacc 
compiler generator for parsing and of the standard Gnu G-k- 1- container li- 
braries for hashing of variable names. Including interfaces to Oz, AMPL, and 



David Abramson, personal communication. 
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Cplex the code is roughly 7000 lines of source code. However, the core al- 
gorithms of Wsat(oip) require only around 2500 lines, and thus represent a 
comprehensible piece of code. 

Incremental Data Structnres. In order to perform the local moves effi- 
ciently, it has often been emphasized for SAT local search that incremental 
data structures can greatly enhance the performance. The fundamental data 
structure used to represent the unsatisfied constraints are two linked arrays 
(used once for hard and once for soft constraints): At any point in time, 
one direction u[l . . .k] maintains the indices of all k unsatisfied constraints 
for efficient selection of a random unsatisfied constraint. The other direction 
ui[l ... to] is used to efficiently update u and maintains the index where con- 
straint i is positioned in the u array if currently unsatisfied, i.e. u[ui[ij] = i, 
illustrated in Figure 3.6. These arrays are updated whenever a constraint 
changes between sat and unsat. 



u 



ui 




[ 1 ] 

[ 2 ] 

[3] 

[4] 

[5] m=5 



Figure 3.6. Linked arrays for efficient constraint selection and updates. Here, 
three constraints (3,2,5) are unsatisfied. 



The bottleneck computation of the local search process, however, is the 
evaluation of the proposed moves. Therefore, the Wsat(oip) implementation 
employs an extension of the incremental data structures used by Walksat and 
Gsat [135], which have been described in detail in [107]. 

The evaluation of the left hand side of each constraint under the current 
assignment (cx) is incrementally maintained. Further, to efficiently compute 
a change of the overall score upon changing a variable value, a list is main- 
tained for each variable v which links v to all the constraints it occurs in 
with non-zero coefficient. To compute the score of changing a variable, it is 
sufficient to visit only the constraints that it appears in, add or subtract the 
local coefficient to the current left hand side, and reevaluate if the constraint 
becomes sat or unsat. After changing the value of a variable, an update takes 
place of the u and ui arrays, and the value of the incremental total score is 
adjusted as before. 

Another important feature of the algorithmic design for constraints with 
many variables is that only those variables are ‘scored’ for which changing 
can improve the score of the selected constraint. This turned out to be a 






3.4 A Graphical Interpretation 



43 



critical extension in a number of case studies with large numbers of variables 
per constraint. 



3.4 A Graphical Interpretation 

To illustrate the search process, its limitations and various ways to improve 
it, we will now look at how Wsat(oip) moves in the space of integer solutions 
in a graphical example. Consider the following OIP (3.12) and its graphical 
equivalent in Figure 3.7, as it is usually depicted for IP/LP problems. 

(A) 9xi + 5x2 > 45 

(B) X1+X2 >6 . . 

(C) 8x1 + 5x2 < 0 (soft) ' 

a;i,X2 G {1,2,. ..,5}. 

One easily verifies that the OIP (3.12) is confined and can thus be reduced 
to an integer linear program with minimization objective 8xi+5x2. Figure 3.7 
shows the feasible region of the LP relaxation shaded in gray, and delimited by 
the constraints (the feasible region of soft constraint (C) is shaded light gray) . 
The black circles are the points that are feasible to the integer problem, the 
white circles are the points that are integer but are outside the feasible region 
defined by the (hard) problem constraints. Also plotted is one isocost line 
(arbitrarily starting at point (3,5)) and denoted by the equation z = 8xi+5x2. 
All isocost lines are parallel to the plotted one, and the intersection of the 
feasible region with the leftmost possible isocost line (minimizing z) yields 
the LP optimum. 

To illustrate the search process of Wsat(oip) and several improvements 
by this example, we will consider in order (a) the search process with vanilla 
parameters, (b) search with tabu tenure t = 1, (c) search with constraint 
weights A, (d) initialization by rounding of the LP optimum, and (e) search 
after constraint LP rebounding. 

The Figures 3.8 illustrate the moves of Wsat(oip) to solve the problem 
(3.12) using different strategies. First, Figure 3.8(a) depicts the progress of 
vanilla Wsat(oip) after an initialization to (0,0). Each diamond plots one 
variable assignment visited during the search. Each arrow plots the trajectory 
between two assignments, and each arrow is marked by the constraint (A,B, 
or C) that is selected for partial repair. If different selected constraints lead 
to the same move, the transition arrow is marked with more than one letter 
(e.g. A,B).9 

To eliminate the random aspect from this discussion, we do not allow 
random moves, i.e. we always start at point (0,0) (setting pzero = 1), always 
take the best possible move {pnoise = 0) and always pick a hard constraint 

® In the particular example we are lucky in that no branching trajectories exist. 
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Figure 3.7. Graphical Interpretation of problem (3.12). 



for repair when one is violated {phard = !)• Otherwise, we employ standard 
parameter settings. Thus, Maxstep is 2, which means that individual variable 
changes are allowed up to two units in this example, leading to arrows of up 
to length 2. 

The vanilla search 3.8(a) starts by greedy moves along the x\ axis, since 
the score improvement is larger along xi than X 2 - For another 3 moves, 
Wsat(oip) continues moving towards the feasible region, first reaching A’s 
feasible region at (5,0), then the overall feasible region and then reaching the 
local optimum (5,1) with an objective function value of 45. Now, both hard 
constraints are satisfied and the soft constraint C is selected for repair. With 
respect to the score gradient, decreasing X 2 appears to be the best alternative, 
reaching the previously visited point (5,0). At this point, the search starts 
cycling because the noise level is zero. 



Tabu Search. The tabu search 3.8(b) proceeds similarly, except that the tran- 
sition back to (5,0) is tabu and the search happens to enter the feasible region 
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(a) Vanilla Wsat(oip). (b) Search process with tabu tenure 

t = 1. 

Figure 3.8. Search trajectories of different Wsat(oip) strategies. 



at the IP optimal point (4,2) with an objective function value of 42. We no- 
tice that although the tabu element prevents cycling in this case, the search 
reaches the optimal solution only after a detour. The tabu mechanism was 
also critical to find good solutions in several case studies. 

Observation of 3.8(a) reveals that the mistake of the vanilla search was 
that it did not turn at (4,0). Instead of moving more directly towards the IP 
optimum, the score suggested that the better move was to achieve feasibility 
of constraint A. Using this example, we will discuss more elaborate techniques 
to alleviate the situation. Some of the following techniques have been applied 
within the application case studies. 



Constraint Weights. In the vanilla search at point (4,0), why did the score 
suggest to move along x\l Clearly, in terms of feasibility, (4,2) is superior 
as it is feasible with respect to both constraints A and B. Nevertheless, the 
violated of soft constraint C dominates and the score indicated to take the 
move (5,0). In this example, however, moving towards the feasible region 
first would have yielded a better solution. One way to achieve this with 
Wsat(oip) is to increase the weights of hard constraints. The trajectory in 
Figure 3.8(c) is the result of setting A to a large integer for A and B, say 100 
(illustrated by bold lines). Constraint weights were also critical in the case 
study on capacitated production planning. 
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(c) Search process with weights on 
hard constraints. 

Figure 3.8. (cont) Search trajectories 




(d) Search process initialized by 
rounding the LP optimal solution o. 

of different Wsat(oip) strategies. 



Initialization by Rounding LP Solutions. Sometimes it is useful to start from 
an initial solution which is close to the LP optimum, as sketched in the 
previous section. Figure 3.8(d) depicts this search process which consists of 
no moves at all, as the initial solution is already IP optimal (which is of 
course rarely the case in practice). Notice that optimality is proved here by 
the LP optimum of 41.25: there exists no integer solution with a cost better 
than 42 and above the LP optimum. 



Constraint Rebounding. In section 3.1.3, we mentioned that it can be advan- 
tageous to relax the threshold of a soft constraint and enlarge its feasible 
region. As we will see, our example is such a case. By solving the LP relax- 
ation, we determine that the soft constraint C can never be fully satisfied 
as the LP optimum is 41.25. Relaxing the right-hand-side of inequality C to 
a valid lower bound is a confined rebounding and yields the system (3.13), 
which has the same set of solutions as the original problem (by proposition 
3.1.3).!° 



10 



Notice that in general there is more than one soft constraint and more com- 
plex rebounding techniques need to be applied if the bounds are not inherently 
provided in the model. 
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(e) Search process after constraint 
LP rebounding. 



Figure 3.8. (cont) Search trajectories of different Wsat(oip) strategies. 
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The search process on the rebounded system is illustrated in figure 3.8(e). 
Through the rebounding, the assignments along the trajectory are now all in 
the feasible region for C’, and the search concentrates on entering the feasible 
region of the hard constraints. Notice that after the rebounding, the objective 
function values are shifted down by 41.25 — 0, thus the optimal IP solution 
is now 0.75. 



3.5 Related Work 

There are three principal lines of related work, (i) General-purpose heuristics 
for integer programming, (ii) local search strategies for constraint satisfaction 
(CSP) problems, and (iii) domain-specific heuristics for problem classes like 
set-covering, generalized assignment, or time-tabling. We will focus on (i) and 
(ii) and will not discuss the numerous domain-specific heuristics (iii) here. 

In the following, we will present a thorough review of the literature on 
general-purpose heuristics and summarize the problem classes used for exper- 
imentation. As one indicator of problem difficulty, the largest problem sizes 
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are additionally reported in the form ^ where n is the largest number 
of variables (‘columns’) and m is the largest number of constraints (‘rows’) 
reported in the experiments. 



3.5.1 Integer Programming Heuristics 

We start by discussing related work on general purpose heuristics for inte- 
ger programming from the OR literature. There are two principal charac- 
teristics of integer programming heuristics. They either relax the integrality 
constraints and operate on continuous variables. Or, they adhere to the inte- 
grality constraints and perform local moves in the space of integer solutions, 
like Wsat(oip) does. 

Integrality Relaxing Heuristics. A pioneering 0-1 integer programming 
heuristic which combines a variety of techniques is the pivot& complement 
heuristic (P&C) of Balas and Martin [9]. It exploits the fact that an optimal 
solution to a 0-1 IP problem can be found at one of the extreme points of 
the linear programming feasible region. 

P&C has both of the above characteristics and operates in two phases. The 
first (“search”) phase performs mainly pivot moves (in the simplex tableau) 
attempting to find at a good feasible 0-1 solution. Once feasibility has been 
achieved, the second ( “improvement” ) phase attempts to improve the solution 
objective by flipping in turn one variable, a tuple or a triplet of variables. Ex- 
perimental results with P&C have been reported in [9] for capital budgeting 
(200x30)^ set covering , set partitioning (crew scheduling). Tabu 

search enhancements of P&C (treating P&C as a black-box subroutine) a 
have been evaluated on multi-constrained knapsack problems [3, 104] 

as well as special set covering and miscellaneous problems [3]. 

Another strategy that superimposes tabu search principles to extreme- 
point transitions has been given by Lpkketangen and Glover [103]. It uses 
advanced tabu search strategies for diversification and a learning approach 
called ‘target analysis’. Experimental results are reported for multi-constrain- 
ed knapsack problems Recently, Glover and Laguna [59, 60] have pro- 
posed a theoretical basis for IP heuristics. Their approach shares a foundation 
with a framework for generating cutting planes for IP problems, but has not 
yet been experimentally evaluated. 

Integer Local Search. The next of kin of Wsat(oip) is the class of integer 
local search heuristics. Among the few strategies which have been reported 
is a simulated annealing strategy (gpsiman) by Connolly [33] as well as a 
refinement (rfsa) and a variation (pisa) thereof by Abramson et al. [4]. rfsa 
adds the search space reduction technique of Section 3.3.3 to GPSIMAN, and 
PISA uses a different (inferior) neighborhood transition approach. 

In essence, gpsiman proceeds as follows. In each iteration, suppose we 
start from a feasible assignment A. A random 0-1 variable is selected and 
flipped. Then, an attempt is made to restore feasibility of the changed 
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assignment, yielding A! . Next, the score C{A) is compared to C{A') and 
the move is always accepted provided it is not deteriorating the score (i.e. 
AC = C{A') — C{A) < 0). If A is worse than A, the move is accepted with 
a probability where T is a temperature that is decreased according to 

an annealing schedule. The annealing schedule starts from a high tempera- 
ture (many deteriorating moves allowed) and is slowly reduced. Occasionally, 
a temporary heheating phase takes place. 

The feasibility restoration of GPSIMAN and rfsa (‘restore feasibility’) pro- 
ceeds by ranking the variables according to a ‘help-score’ and choosing the 
best variable. Unlike Wsat(oip), the computation of the help-score does not 
predict the true effect of flipping a variable. Instead, it computes an intricate 
measure of the repair effect, taking into account a ‘criticality’ of the violated 
constraints. Details of the computation can be found in [4] or [33]. In contrast, 
PISA (‘penalize infeasibility’) takes a different approach by incorporating vio- 
lated constraints into the objective function, and not restoring feasibility after 
each flip. Similarly as Wsat(oip), pisa allows the search to move through 
infeasible regions of the search space. Abramson et al. do not commit to a 
particular penalty function but find PISA to have inferior performance than 
RFSA. In [4], a number of disadvantages of this scheme are discussed that do 
not apply to Wsat(oip), e. g. “a new cost function is required every time a 
new problem is encountered.” The version of GPSIMAN we had available is 
implemented in Fortran. 

Wsat(oip) is related to both PISA and RFSA: Although Wsat(oip) can 
move through infeasible regions, its repair strategy {phard) be set to 
immediate feasibility restoration as well. We see the principal differences to 
GPSIMAN in the score computation of Wsat(oip), the applied principles from 
tabu search, and the fact that Wsat(oip) operates on OIPs according to the 
Walksat-Principle . 

Experimental Comparison with GPSIMAN. We have evaluated the GPSIMAN 
solver on our benchmarks (rfsa has ceased service and was not available). 
However, with exception to the radar surveillance problems (Section 6.1), it 
did not succeed in finding acceptable solutions. For the radar surveillance 
problems, the results were not competitive to those of Wsat(oip), despite 
GPSIMAN was allowed more than two orders of magnitude the runtime of 
Wsat(oip). The results are consistent with previously reported experiments 
by Abramson et al. [4]. 

For the course assignment problems, GPSIMAN was not able to find sat- 
isfactory solutions to even small instances. Also, the timetabling problems 
in Chapter 5 (all tightness levels) were beyond the limitations of GPSIMAN, 
which gives supporting evidence for Abramson et al.’s conclusion that “al- 
though the RFSA approach performs better than the PISA approach, it fails 
for problems that are heavily constrained”. GPSIMAN does not support gen- 

Unfortunately, the implementation produced illegal (super-optimal) solutions 

when given more promising parameters. 
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eral integer variables and therefore not applicable to the Chapter 7 problems. 
In [33] GPSIMAN experiments are described on quadratic assignment ^ , 

grids-and-crosses knapsack processor-communication (sex 698)^ 

graph coloring and class timetabling rfsa and PISA have 

been evaluated and compared on large set partitioning problems (’’sissx492) 
using extensive preprocessing and dynamic search space reduction [4]. 

A Simulated Annealing Code. A recent approach, intsa, by Abramson and 
Randall [5] combines different neighborhood transition schemes of simulated 
annealing for different combinatorial problems, intsa is reported to perform 
well in comparison with GPSIMAN and a IP branch-and-bound solver (OSL). 
The goal of the framework is to automatically choose an appropriate SA 
neighborhood based on the given algebraic problem description. For example, 
choose value changes for graph coloring, but choose a 2-opt move for the type 
of constraint used to represent traveling salesman problems. 

While this is clearly an appealing idea, it remains to be shown that intsa 
will be able to handle problems with mixed types of constraints (in addition 
to the pure problems handled by problem-specific SA implementations). In 
fact, as [5] note, “the intsa results were actually gathered from a number 
of different programs, each of which handled one of the classes of constraints 
rather than one program which could differentiate the constraint class and 
choose the appropriate algorithm.” intsa has been evaluated on quadratic 
assignment (30 facilities), traveling salesman (666 cities), graph coloring (300 
nodes, 740 edges), bin packing (500 items, 201 bins), and generalized assign- 
ment (8 agents, 32 jobs). 

We would very much like to see an approach that combines the strengths 
of intsa and Wsat(oip): different neighborhood schemes combined with 
flexibility in the type of constraints. One possible scenario would be to employ 
the Walksat Principle for dispatching the repairs to be made. 

CSP Tabu Search as a General Problem Solver. Very recently, an indepen- 
dent approach of iterative repair to linear integer constraints was published by 
Nonobe and Ibaraki [116] (previously presented at APORS-97 in Melbourne, 
December 97; Wsat('P,B) appeared at AAAI-97). It shares with this work the 
iterative repair approach to ILP problems, and also provides a comprehensive 
empirical study with encouraging results. Technically, the approaches are less 
similar. First, the framework by Nonobe and Ibaraki does not start from SAT 
local search, and hence uses different local moves (‘shift’ and ‘swap’) instead 
of the atomic flip moves performed by Wsat(oip). Further, to approach op- 
timization problems, [116] introduces a mechanism to tighten a bound on the 
objective function and an additional control mechanism. This is in contrast 
to our use of OIP for this purpose. Further, [116] employs an open definition 
of CSP as base representation and hence does not address combinations with 
linear programming. Also, [116] employs solely 0-1 variables, and while two 
of the experimental studies are on similar problem types, no application to 
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tight 0-1 ILP feasibility problems seems to be provided in [116]. It would be 
interesting to further compare the two approaches. 

3.5.2 Local Search in Constraint Satisfaction 

The other line of work on domain-independent local search has taken place 
in the context of constraint satisfaction problems (CSPs) in artificial intel- 
ligence. For the purpose of the discussion, we distinguish between binary 
CSPs (in which all constraints involve exactly two variables) and non-binary 
CSPs. Also, we distinguish between an extensional representation (in which 
constraints are represented by an explicit set of allowed or forbidden variable- 
value tuples) and intensional representations. Of course, algebraic constraints 
are non-extensional and generally non-binary representations. 

One of the earliest approaches of heuristic search in constraint satisfac- 
tion is the min- conflicts heuristic by Minton et al. [109] (previously published 
1990). It has been formulated both as a backtracking algorithm and as a hill 
climbing strategy. The basic principle is the same as in SAT local search, 
namely to “select a variable that is in conflict, and assign it a value that min- 
imizes the number of conflicts” [109]. Min-conflicts was evaluated on graph 
coloring (binary), the n-queens problem (binary), and scheduling of the Hub- 
ble Space Telescope (non-binary). 

Different methods in the same spirit were evaluated for graph coloring and 
frequency assignment by Hao and Dome [68] . The min-conflicts strategy was 
enriched with noise and applied to randomly generated binary MAX-CSP 
problems by Wallace and Freuder [147]. 

Another line of work is the connectionist approach GENET by Wang, 
Tsang, Davenport et al. [38, 39]. GENET is an iterative repair network ap- 
proach that operates by a heuristic learning rule. Although GENET was 
initially formulated for binary extensionally represented GSPs, various ex- 
tensions to non-binary non-extensional GSPs have taken place, among them 
the work on E-GENET [99, 100]. In E-GENET, more expressive constraints 
have been studied (each on one type of benchmark): A ‘linear-arithmetic’ 
constraint (cryptarithmetic puzzles), an ‘atMost’ constraint (car sequencing), 
a ‘disjunctive’ constraint (Hamiltonian path), and a ‘cumulative’ constraint 
(simple scheduling problem). 



3.6 Summary 

In this chapter, we have introduced Wsat(oip), an integer local search 
method which operates on an algebraic problem representation. Wsat(oip) 
generalizes Walksat, a successful local search procedure for propositional sat- 
isfiability (SAT), to more expressive constraint systems. 

For the purpose of the generalization, we have introduced over-constrained 
integer programs (OIPs), a constraint class which is closely related to integer 
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programs. OIP allows for a natural generalization of the principles of SAT 
local search to integer optimization. Further, it has been shown that OIPs are 
a special case of integer linear programs and permit combinations with linear 
programming for bound computation, initialization by rounding, search space 
reduction, and feasibility testing. The representation is similar enough to in- 
teger programs to make use of existing algebraic modeling languages as front- 
end to a local search solver. To improve performance on realistic problems, 
Wsat(oip) incorporates strategies from Tabu Search. On a graphic example, 
we have illustrated different sub-strategies for local moves in Wsat(oip), i. e. 
a tabu element, constraint weights, initialization by rounding and constraint 
rebounding. 
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“Any choice of [benchmark] problems is open to the criticism that it is un- 
representative. There is another way, however. One can investigate how algo- 
rithmic performance depends on problem characteristics. The issue of problem 
choice, therefore, becomes one of experimental design. Rather than agonize 
over whether a problem set is representative of practice, one picks problems 
that vary along one or more parameters." 



John N. Hooker in [75] 



The previous chapter has described new methods for integer local search 
and presented the Wsat(oip) procedure. The next three chapters will em- 
pirically investigate the performance of Wsat(oip) in a number of realistic 
case studies. In between, this chapter reflects on issues and goals of our ex- 
perimental analysis. 

There are three chapters for three different aspects of Wsat(oip): Chap- 
ter 5 investigates Wsat(oip)’s ability to solve difficult 0-1 integer feasibility 
problems. Chapter 6 concentrates on 0-1 integer optimization problems. Fi- 
nally, Chapter 7 focuses on the extension of Wsat(oip) to (non 0-1) finite 
domain problems. 

There can be different goals of experimental analysis [124, 75, 11] . Because 
integer local search is at an early stage, our case studies mainly investigate 
two central aspects: Domain-independence and applicability to optimization 
in practice. To demonstrate domain-independence, we draw applications from 
a range of integer optimization problems. To support the claim of practicality, 
we highlight the aspects that we believe matter for practical concerns. This 
chapter contemplates criteria of success for practical optimization methods 
and motivates the case studies on the grounds of those criteria. 



4.1 Optimization in Practice: Criteria of Success 

Theoretical analysis of algorithms is usually concerned with aspects of worst- 
case or average case resource requirements. Normally, one is interested in 
bounds on resource usage such as time or memory, usually under varied prob- 
lem parameters such as size. Additionally, for approximation algorithms, one 
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attempts to derive lower bounds on the quality of the solutions. However, 
at present most practical optimization algorithms for IVP-hard problems are 
beyond the scope of rigorous theoretical analysis [75, 86, 11], even worse so 
when applied to realistic problem classes. In this situation, one needs to resort 
to experimental testing. 

When moving from theoretical to experimental analysis, one is faced with 
an unfamiliar amount of freedom. Factors that often limit theoretical anal- 
yses disappear, such as restrictive assumptions on the instance distribution 
or restrictions on the algorithmic properties to investigate. Hence, careful 
decisions need to be made what aspects an empirical evaluation should in- 
vestigate. The central decisions are how to define algorithmic performance 
and to select a set of ‘typical’ problems to evaluate. 

Normally, given a problem instance, performance is measured in terms 
of (i) time to obtain the first or best solution, or (ii) best solution quality 
obtained in limited time. Orthogonally, however, it is critical to investigate 
the variation of these measures on a given instance distribution. We refer to 
performance variation over a given instance distribution as robustness . Note 
that there is an another issue of robustness, namely performance variation 
when changing the instance distribution (i. e. considering different problems), 
which we call flexibility. 

On first sight, robustness might be regarded as a question of secondary 
importance in comparison to solution quality or runtime. But in fact, it is 
of critical importance and inseparable. In particular for NP-hard problems, 
an algorithm that performs well on one set of problem instances is hardly 
of any practical use if small variations of the instance parameters break it 
(if its runtime changes from 4.83 seconds, precisely measured on one given 
instance, to 2 weeks on the next). 

There are several aspects of robustness. First, we are interested in the 
scaling of runtime with increasing problem size (keeping other problem char- 
acteristics similar). The second aspect is scaling of runtime with increasing 
constrainedness, i. e. how does performance vary as additional constraints are 
thrown at the problem.^ A third aspect is residual robustness, i. e. robustness 
under minor variations of instance characteristics which only remotely affect 
size or constrainedness. 

4.1.1 Scaling with Increasing Problem Size 

At the center of most theoretical algorithm analysis is the question how 
an algorithm’s performance varies with increasing problem size. Despite the 
importance commonly attributed to this question on a theoretical level, it is 
sometimes neglected in experimental studies. 

Sometimes, there are good reasons not to study scaling with size, espe- 
cially on real problem instances, where one usually has no handle on the size 

^ We will not attempt to give a rigorous definition of ‘constrainedness’. 
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of the problem. Even if instance size varies, real problems sometimes happen 
to vary strongly along a number of characteristics and size may appear as ‘.just 
another parameter’. It is then difficult to isolate size from the other parame- 
ters. However, if artificially generated problems are studied, there is normally 
no reason not to investigate scaling (care must be taken not to change the 
problem characteristics when crafting instances of different size). We measure 
problem size by the number of variables and constraints of a given encoding. 
The scaling behavior is of practical importance since real problems are often 
large — typically at least as large as state-of-the-art technology can handle. 

Moreover, what makes scaling critical is that different algorithms exhibit 
different scaling behavior. Empirically, what is a excellent algorithm for small 
problems may not be applicable to large-scale problems. Conversely, a heuris- 
tic that works well for large problems may not have the desired properties for 
small problems (i. e. because it is approximate and one might care for optimal- 
ity) . Scaling of local search has previously been examined on hard randomly 
generated satisfiability problems and sub-exponential (average) scaling was 
observed [53, 118]. In our case studies, we examine the scaling behavior of 
integer local search for realistically structured (randomly generated) covering 
problems as well as for real course assignment problems in Chapter 6. 



4.1.2 Scaling with Increasing Constrainedness 

A second dimension in scaling occurs with increasing constrainedness. Inves- 
tigating algorithmic behavior along this dimension is important in particular 
as the typical practitioner’s approach to constraint problems is iterative re- 
pair: “State some known constraints and find a solution by invoking a solver. 
Observe that it exhibits certain unliked characteristics and state additional 
constraints that disallow them. Re-solve and iterate.” In this typical spiral 
process, it is critical that both the loosely constrained and the more tightly 
constrained problem can be solved. 

In recent years, there has been some interest in AI in studying the al- 
gorithmic performance across different degrees of constrainedness. For sev- 
eral problem domains and algorithms, an easy-hard-easy pattern has been 
observed in time-to-first-solution as the problem constrainedness is being in- 
creased. Most of these studies have investigated performance on randomly 
generated problem instances. As yet, there is no generally accepted way to 
quantify the constrainedness of a problem instance, although measures for 
the constrainedness of an ensemble have been given [51, 110, 151]. 

Chapters 5 (timetabling) and 7 (production planning) shed some light on 
the issue of scaling with increasing constrainedness on real problems. While 
in the first study we use the number of solutions as a rough measure of con- 
strainedness, the latter uses a parameter of the input problem, i.e. resource 
capacity. 
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4.1.3 Flexibility and Residual Robustness 

Another requirement of practical optimization methods is flexibility. When 
a method is specifically tailored to a narrow problem class (e. g. set cover- 
ing or generalized assignment or time-tabling etc.), incorporating additional 
constraints to solve a closely related problem usually requires adjusting the 
algorithms or replacing the strategies altogether. Flexibility is investigated 
both across the different case studies (if a method is domain-independent, it 
will have to be ffexible) and within the case studies (each of the considered 
problems includes a variety of different constraints). 

The last aspect of runtime variation is residual robustness: After factoring 
out issues of size and constrainedness, we are left with a residuum of perfor- 
mance variation on a given instance distribution. For real problems, however, 
it may be difficult to obtain several instances with similar parameters. One 
solution is to perturb the input parameters of a given instance, thereby gen- 
erating ‘pseudo-real’ problems.^ We refrained from perturbing real problems 
to avoid the difficulty of choosing which factors to perturb. In most cases, 
the case studies examine several similar instances of a given problem. 



4.2 The Problem Class Selection 

In recent years, the use of randomly generated benchmark problems has in- 
creasingly been criticized for empirical evaluation of algorithms, e. g. [88]. To 
address the need for a realistic assessment of optimization technology, our 
choice of benchmark problems focuses on problems ‘as real-world as avail- 
able’. All benchmark problems have either been studied in the recent AI or 
OR literature (timetabling and sports scheduling), are the result of indus- 
trial cooperations (radar surveillance covering and capacitated production 
planning) or originate from operating applications (course assignment, sports 
scheduling) . 

The benchmarks have been selected to examine the four requirements 
for practical optimization methods stated above: (i) scaling with problem 
size and (ii) constrainedness, (iii) flexibility and (iv) residual robustness. All 
benchmark problems under consideration share that they stem from NP-hard 
problem classes, have a large number of variables and constraints, contain a 
heterogenous set of constraints, and are difficult for the best general state- 
of-the-art optimization techniques available (both scientific and commercial 
packages) . The set of benchmarks diverges from the classical pure problems 
(e.g. set covering, set partitioning, generalized assignment etc.) and involves 
complicating side constraints which, in most cases, would have prevented a 
direct application of domain-specific heuristics from the literature. All case 
studies are on integer linear problems. All but one case study are on 0-1 
integer optimization (binary variables). 

^ A term used by Toby Walsh in personal communication. 
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Two case studies (Chapter 5) address the ability to solve hard feasibility 
problems, a property not commonly addressed by optimization benchmarks. 
Such problems are typically difficult for general-purpose integer optimiza- 
tion methods like IP branch-and-bound. We did not investigate graph col- 
oring problems because several studies of iterative repair exist for this do- 
main [135, 68, 109]. Clearly, integer optimization problems vary largely and 
the particular selection only covers a small fraction. The claim on domain- 
independence should hence be viewed in relation to the state-of-the-art ILP 
solver technology. 

The benchmark problems have the following size characteristics (maximal 
number of variables and constraints): Radar surveillance covering (10989 x 
14595), course assignment (8404x11350), the Progressive Party Problem 
(4632x30965), ACC Basketball (1339x3053), and production planning (us- 
ing finite domain variables, 7520x3047). Although problem size does not 
directly imply hardness, it is a relevant problem characteristic and we notice 
that the size of the benchmarks is larger than in many previous studies of 
general-purpose methods. 

With exception to capacitated production planning (which contains pro- 
prietary data), all benchmark problems have been made available through 
the Constraints Archive at http://www.cirl.uoregon.edu/constraints/. 



4.3 The Empirical Comparisons 

To demonstrate the performance and range of applicability of integer local 
search, this study takes a competitive approach to performance evaluation. 
It focuses on comparing Wsat(oip) to other general-purpose optimization 
frameworks, which have been described in Chapter 2. Whenever appropriate, 
we thus compare to IP/MIP branch-and-bound (Cplex 5.0 [80]), constraint 
programming solvers (Oz [137] and ILOG Solver [79] approaches from the lit- 
erature), and GPSIMAN, a domain-independent simulated-annealing heuristic 
[33, 4].^ The employed constraint programming approaches incorporate some 
domain knowledge in the form of enumeration heuristics or suitable problem 
factorizations. 

In order to compare the results of the Wsat(oip) heuristic to exact algo- 
rithms, the exact methods are run in “heuristic mode” [9], i. e. the best solu- 
tion found within a given time limit is reported (occasionally, second-best so- 
lutions are also reported if qualities are similar but time differs significantly), 
if optimality can be proved, this is reported. To evaluate the experimental 
results in absence of provably optimal solutions, we employ methods for gen- 
erating valid lower bounds, i.e. linear relaxation and Lagrangean relaxation. 



® I thank David Abramson and Marcus Randall for providing the GPSIMAN solver 
of David Connolly and for useful suggestions to parameter settings. 
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For the MIP branch-and-bound experiments, the Cplex 5.0 MIP opti- 
mizer [80] is used as it is commonly regarded as one of the fastest general- 
purpose MIP-optimizers and has been in commercial use for over 10 years. 
Cplex 5.0 utilizes state-of-the-art algorithms and techniques, including cuts 
(cliques & covers), heuristics, a variety of branching and node selection strate- 
gies, and a sophisticated mixed integer pre-processing system [80] . We often 
run Cplex with standard parameters, which involves automatic control of 
several MIP subroutines (such as heuristics, branching, cut generation) . The 
standard parameter settings are usually non-trivial to improve upon (in many 
cases we report on non-standard settings as well) . It should be recognized that 
Cplex is the product of several man-decades of development and research, 
whereas the Wsat(oip) implementation is comparatively simple. 

Mostly, run-times are reported but no memory requirements even though 
in some applications space usage may be an important issue. We neglect 
the issue in our study, but point out that the memory usage may differ 
significantly for the different frameworks: Tree-search approaches like Cplex 
and CP sometimes occupy hundreds of megabytes of main memory while 
integer local search uses constant memory during the search. To compensate 
for exaggerated memory usage, all runtimes are measured as wall clock time, 
which purposefully incorporates a penalty for paging. 
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This chapter investigates two difficult time-tabling/scheduling problems, ‘the 
Progressive Party Problem’ (PPP) and scheduling of the Atlantic Coast (bas- 
ketball) Competition of 1997/98 (ACC). Both problems were recently intro- 
duced and solved in the literature [115, 136]. The previous results demon- 
strate that finding feasible solutions for these problem is challenging, even 
when using approaches that incorporate domain-knowledge. 

Both problems can be encoded with 0-1 integer constraints (a model for 
PPP has been given in [136] and a model for the full ACC problem will be 
presented here), but no approaches have previously been reported to find 
solutions directly from 0-1 encodings of these problems. 

This chapter reports on experiments of integer local search given 0-1 in- 
teger models of problem instances of PPP and ACC. For both problems, 
the experimental results will be compared to the previously reported results. 
Moreover, we will study the performance of local search with increasing prob- 
lem constrainedness. In particular, for the ACC problem, an extensive study 
of integer local will be presented that investigates the scaling of runtime with 
increasing problem constrainedness. 



5.1 The Progressive Party Problem 



The problem in the first case study, “the progressive party problem”, was 
recently introduced in a comparison between constraint programming and 
integer linear programming [136]. A main result of the study is that the 
problem appears to be beyond the size limitations of integer linear program- 
ming (ILP) but can be solved using constraint propagation and chronological 
backtracking. Our experiments show that the problem can be solved signif- 
icantly faster using Wsat(oip). Further, we look at slight variations of the 
instance given in [136] and find that local search is robust with respect to 
the modifications. On the other hand we were not able to find a constraint 
program that could solve all of our test problems. To solve the problem with 
Wsat(oip), we factor it into two stages. In the first stage, a small number 
of principal variables are explicitly enumerated (e. g. using constraint pro- 
gramming), while in the second stage, the variables valued in stage one are 
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Table 5.1. Boat specifications. The entries are boat number i, spare capacity 
Ki — a and crew size d. 
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propagated through the theory, and the remaining subproblem is attacked 
with local search. 



5.1.1 Problem Description and Formulation 

In the integer local search approach, we employ a 0-1 model similar to the 
one used by Smith et al. [136]. The problem model is large and incorporates a 
variety of different constraints which suggested that it would be an interesting 
test case for integer local search. 

The problem scenario is an evening party in the context of a yachting 
rally. Certain boats are selected to be hosts, and the crews of the remaining 
boats in turn visit the host boats for several successive half-hour periods. 
The crew of a host boat remains on board to act as hosts while the crew of 
a guest boat together visits several hosts. Every boat can only host a limited 
number of guests at a time and crew sizes are different. Table 5.1 reports boat 
capacities and crew sizes. There are six time periods. A guest boat cannot 
revisit a host and guest crews cannot meet more than once. The problem 
facing the rally organizer is that of minimizing the number of host boats 
(presumably for reasons of supply logistics): Certain boats are constrained 
to be hosts, and selecting the hosts among the remaining boats is stated as 
part of the problem. 

We do not claim that this problem is of immediate practical significance; 
however, it has the advantage of being a well-studied hard time-tabling prob- 
lem with a variety of constraints. The variables in the problem are the fol- 
lowing: = 1 iff boat i is used as host boat. Variables jikt = 1 iff boat k is 
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a guest of boat i in period t. Constant c, is the crew size of boat i and Ki 
is its total capacity. The objective is to minimize the number of hosts 5i, 
subject to: 

Constraints CD. A boat can only be visited if it is a host boat. 

likt — < 0 for all i, k,t', i ^ k. 

Constraints CCAP. The capacity of a host boat cannot be exceeded. 

Ckjikt < Ki - a for all i, t. 

k,k^i 

Constraints CA. Each crew must always have a host or be a host. 

4 + ^ likt = 1 for all k, t. 

i,i^k 

Constraints CB. A guest crew cannot visit a host boat more than once. 

likt < 1 for all i,k] i^ k. 

t 

An additional set of 0-1 variables was introduced to state the meet-once 
restrictions, mkit = 1 if boats k and I meet at time t. This simplifies the ILP 
model described in [136].^ 

Constraints U. Link mut with likt- 

hkt + hit - rrikit < 1 for all k, l,t; k < 1. 

Constraints M. Every pair of hosts can meet at most once. 

vrikit < 1 for aA k,l\ k < 1. 

t 

With B boats and T time periods, the problem has 0{B^T) variables 
and 0{B^T) constraints in this formulation. Smith et al. note that the CP 
representation is more compact and has “far fewer constraints and variables 
than the ILP” . This is not the case since the number of both constraints and 
variables is actually 0{B^T) in both encodings (even in the improved ILP 
model in [136]). 

Although the problem is formulated as an optimization problem, given 
the particular description of the participating boats the task is to find a 

^ The original ILP description [136] is mkit = 1 iff boats k and I meet at time t. 
The modification simplifies the problem and saves approximately 30K clauses. 
According to Sally Brailsford (personal communication) this had been tried in 
the ILP model. 
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feasible assignment with 13 host boats. Every solution with 13 hosts is optimal 
because the capacity constraints cannot be met with 12 hosts even for a single 
time period. Solving the problem can be divided into two stages: (i) selection 
of the host boats, and (ii) assignment of guest boats to hosts for all time 
periods. It turns out that the spare capacity of the boats is a good indicator 
of whether a boat should be host or guest, so after forcing special boats 
to be hosts (e.g. the rally organizer), the remaining hosts were selected by 
decreasing spare capacity (the spare capacity of a boat is its total capacity 
minus its crew size) . In both the ILP and the CP approach. Smith et al. treat 
both stages of the problem. However, the search-space for a particular host 
selection is too large to be explored exhaustively within hours of computation. 
This shows that solving stage (ii) by itself is a hard subproblem and we 
therefore focus on stage (ii): Finding a guest allocation given a fixed selection 
of hosts. Thereafter we will outline a strategy that captures both stages. 

Smith et al. report the problem could not be solved with a commercial 
integer programming tool (XPRESSMP, using a variety of tricks) because it 
appears to be beyond the size limitations of ILP. 



5.1.2 Experimental Results and Comparison 

For the experiments, we use the original problem instance of Smith et al. and 
randomly vary the host selection to produce 5 additional instances. For all 
instances, we keep the original description of boat capacities and crew sizes. 
After fixing the 13 hosts and performing constraint propagation as an efficient 
preprocessing, the original problem has 4632 variables and 30965 remaining 
clauses in pseudo-Boolean formulation. Wsat(oip) finds a feasible guest al- 
location in 5.5 seconds (averaged over 20 successful runs on a SPARCstation 
20) using a tabu memory of size 1 and initializing with a bias of pz = 0.9. Ad- 
ditionally, setting up the constraints from an abstract representation requires 
around 15 seconds. Table 5.2 summarizes the results. 

For comparison. Smith et al. report 27 minutes of runtime of their ILOG 
Solver program on a SPARCstation IPX. To reproduce the results, we im- 
plemented the described modeling in Oz,^ a concurrent constraint language 
[137].^ We used constraints and a labeling strategy similar to the one de- 
scribed by Smith et al.. Although our constraint program was able to solve 
the original instance in 8 minutes (on a SPARCstation 20), we could not find 
a labeling strategy that was able to solve all sample instances.^ 

It has been claimed [136] that the progressive party problem can be solved 
with constraint programming in a straightforward way. Our experiments con- 
firm this for the original problem instance, but we find that slight variations 



^ I thank Jorg Wiirtz and Thorsten Olgart for modeling the progressive party 
problem in Oz. 

® Publically available from http:://www. ps.uni-sb.de/oz/. 

^ Jorg Wiirtz, personal communication. 
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Table 5.2. Experimental results for variations of the Progressive Party Prob- 
lem. The columns are: Selected hosts, total sum of host spare capacities h, 
total sum of guest crew sizes g] percentage of total capacity used as a mea- 
sure of constrainedness (%cap = g/h). Runtimes averaged over 20 runs of 
Wsat(oip), Maxmoves= oo, flip-rate 1.1 K-flips/s. 
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3-13,25,26 
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.96 


8.8s 


1-11,19,21 


95 


93 


.98 


31.6s 


1-9,16-19 


93 


91 


.98 


42.5s 



can make the problem too difficult to solve in hours of computation.® Because 
CP is not a particular algorithm but subsumes a wide variety of techniques to 
operationalize constraint solving, no general conclusion can be drawn about 
its performance on the particular problem. Stronger propagation, better la- 
beling, randomization [63] or search strategies might be able to improve the 
performance robustness on this problem. 

The authors of [136] also report an integer programming approach (using 
XPRESSMP) given the stage (ii) problem, in which a problem with up to 15 
boats and 4 time periods could be solved. 

Embedding into constraint programming. To solve both stages of the prob- 
lem, we propose a loose coupling of systematic and local search. The ap- 
proach simply enumerates the principal variables heuristically (in this case 
the Si’s, stage (i)), then performs constraint propagation/simpliflcation and 
applies local search to solve the remaining subproblem (stage (ii)). In our 
implementation, we use an embedding of Wsat(oip) into the constraint lan- 
guage Oz. The advantage of using a constraint language is the high-level 
support for problem modeling and solution checking. Oz additionally offers 
the use of computation spaces which simplifies the embedding of a solver like 
Wsat(oip) into CP. 

Notes on the Experiments. Before using the two-stage approach, we ex- 
perimented with local search on a version of the problem that included host 
selection. Observation of the local search process revealed that host selection 
and guest allocation were mixed and the host selection was changed almost 
as often as the guest allocation, which seemed to be an unreasonable strategy. 

Before introducing the mkit variables and U, M constraints we solved 
Smith’s first encoding (constraints S,V,Y) with local search. With 32136 vari- 

® We thank Mats Carlsson for confirming this observation with a SICStus FD 
implementation. 
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ables and 90844 constraints (after fixing the hosts), this encoding was much 
larger. Nevertheless, Wsat(oip) solved it in a few minutes. 

Related Work. Recently, Hooker and Osorio [76] introduced a framework 
called Mixed Logical/Linear Programming (MLLP). They apply MLLP to 
the progressive party problem and compare the experimental results to a 
MIP encoding solved with Cplex. Instead of using the two-stage factoring 
presented above. Hooker and Osorio encode both stages of the problem using 
a compact representation with the number of host boats as minimization 
objective. Their approach (MLLP) can solve problems of up to 10 boats and 4 
time periods to optimality within several hours (for comparison, the original 
problem has 29 boats and 6 time periods); MILP is reported [76] to find 
optimal solutions up to 8 boats and 4 periods without manual intervention. 



5.2 The ACC Basketball Scheduling Problem 

In the second case study in timetabling/scheduling, we investigate a diffi- 
cult problem from sports scheduling that was recently studied and solved by 
Nemhauser and Trick [115], the scheduling of the Atlantic Coast Competition 
in basketball {ACC Basketball 97/98). The previous approach by Nemhauser 
and Trick (N&T) involved a domain-specific problem factorization together 
with a mix of integer programming and explicit enumeration leading to a 
solution that was accepted by the ACC. 

Here, we investigate an integer local search approach to the ACC problem 
that works directly from a monolithic 0-1 integer linear program and includes 
all of the documented constraints [115, 141] of the original problem. 

With respect to the experimental results, integer programming and ex- 
plicit enumeration have been reported [115] to find a set of schedules within 
around 24 hours on a modern workstation (the approach is exact and finds 
all solutions to the problem modulo certain restricting assumptions, i. e. the 
particular mirroring scheme).® More recently, a very efficient approach to the 
problem has been reported by Henz [69] that applies constraint programming 
to a problem factorization by Schreuder [130] (similar to the factorization 
used in [115]). 

In contrast to the previous approaches, the integer local search approach 
uses no problem factorization, but still finds solutions that are competitive 
with the official timetable [115] with respect to several optimization criteria. 
In summary, the results of the case study are: 

(i) Both modeling and solving of the ACC problem can be accomplished 
using a monolithic IP representation. Requiring solutions to be at least 



Note that much shorter times have been reported for finding a first solution 
(Michael Trick, personal communication). In the last stage, 300 million schedules 
are generated and filtered. 
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as good as the official timetable with respect to all optimization cri- 
teria given in [115], solutions are found in 30 minutes (on average) by 
Wsat(oip). This is an exciting result for local search as the problem 
(with a potential search space of 2^^^^ variable assignments) has only 87 
solutions! 

(ii) A general purpose heuristic, Wsat(oip), can solve a real instance of a 
dense double-round-robin (DDRR) scheduling problem. 

(iii) In a double round robin competition, the second half of the schedule 
typically mirrors the first. However, to comply with given team pairing 
requirements, such perfect mirroring is not always possible: To handle 
additional pairing requirements, the previous approaches based on fac- 
torization resort to swapping slots of the schedule, which is not possible 
when conflicting pairing requirements exist. To deal with conflicting pair- 
ings, we present minimal distortion mirroring, a new approach in which 
only few pairings are swapped while the basic mirroring scheme is pre- 
served. 

5.2.1 Double Round Robin Scheduling 

In a Double Round Robin {DRR) sport competition, which is a popular 
scheme in many sports, every team t plays against every other team ex- 
actly twice during the competition, once at home (the place of t) and once 
away. 

There are two types of sports schedules: temporally dense and temporally 
relaxed. In temporally dense double round robin scheduling (DDRR) like the 
ACC competition, the number of slots (time periods in which games may take 
place) is almost equal to the number of games that each team must play. If 
there is an even number n of teams, a DDRR schedule has 2{n — 1) slots. If 
n is odd, there are 2n slots in which n — 1 teams play and one team is bye. 

Contrary to temporally relaxed schedules where local improvement heuris- 
tics appear to be used frequently (e. g. [45]), a brief survey in [115] attests 
for temporally-dense schedules that “while some local improvement heuris- 
tics have been found, they tend to be rather limited in scope and heavily 
dependent on finding good initial solutions.” Hence, by using a (i) domain- 
independent (ii) heuristic which starts from (iii) a random initial solution, 
our approach takes several steps in one. 

As with most other real sport scheduling problems, the ACC 1997/98 
problem is constrained by a wide variety of requirements and objectives. For 
example, it is desired to have a large separation between the two games of a 
pair of teams. If the criterion is to minimize the maximal distance between 
any two teams, the minimal separation is half the number of teams. In this 
situation, the problem is typically simplified by requiring that the pairings in 
the first and the second half be identical, except that the places of the games 
are reversed. If the timetable meets this condition, it is said to be perfectly 
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mirrored. In the ACC 97/98 problem, individual team pairing constraints 
prohibit a perfect mirroring. 

Another important aspect concerns the satisfaction of the local spectators 
who prefer not too few and not too many local games in a sequence. There- 
fore, the succession of home and away matches needs to be altered frequently. 
The requirements of the ACC given in [115] with respect to the allowed se- 
quences of home/ away and bye are intricate and rule out a direct application 
of previous work on double round robin tournaments [25, 130]. Further, there 
are constraints that no team should face the particularly strong teams in im- 
mediate succession. 

The third aspect concerns broadcasting, as television networks require a 
stream of “high quality” games and have additional requirements when the 
most popular pairings should occur. Since teams return home after almost 
every away game in the ACC, there are no travel constraints. 

Because the problem characteristics change if details of the specification 
are omitted, it is unavoidable to present the entire list of constraints (as 
will be shown, simplified versions of the problem are in fact very easy for 
integer local search). The following section therefore presents the complete 
“laundry-list” [115] of constraints. 



5.2.2 Problem Specification of ACC97/98 

The ACC in basketball consists of nine universities: Clemson (Clem), Duke, 
Florida State (FSU), GeorgiaTech (GT), Maryland (UMD), North Carolina 
(UNC), North Carolina State (NCSt), Virginia (UVA), and Wake Forst 
(Wake). The problem is to find a 18 slot DDRR for the period of nine weeks 
(12/31/97, a Wednesday to 3/1/89, a Sunday), such that in each week there 
is a weekday and a weekend game. In the following description of the re- 
quirements, we follow exactly the presentation of Henz [69] and distinguish 
‘requirements’ from ‘optimization criteria’. The requirements can be viewed 
as a minimal set of constraints to satisfy, whereas the optimization crite- 
ria may be met in different ways since they are generally in conflict. The 
requirements are the following: 

RO. Double round robin. The teams play a temporally dense double-round 
robin competition. 

Rl. Return match separation. The teams wish return their games as separate 
as possible (i.e. if a at 5 in slot i, then 6 at a at i -I- D for suitably large 
D). The measure is to maximizing the minimum distance. The requested 
separation between games between two teams also holds for byes: No 
team wants its two byes too close together. The minimal temporal dis- 
tance between first leg and corresponding return match must be 7 slots. 
Considering that UNC plays Duke in slot 11 an 18 (see requirement 9 
below) 7 is the maximal value for this minimal distance. 

R2. No two final aways. No team can play away in both last slots. 
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R3. Home/ Away /Bye pattern constraints. No team may have more than two 
away matches in a row. No team may have more than two home matches 
in a row. No team may have more than three away matches or byes in a 
row. No team may have more than four home matches or byes in a row. 
Similar conditions hold for consecutive weekend slots. No team may have 
more than two away matches on subsequent weekends. No team may have 
more than two home matches on subsequent weekends. No team may 
have more than three away matches or byes on subsequent weekends. No 
team may have more than three home matches or byes on subsequent 
weekends. 

R4. Weekend pattern. Of the weekends, each team plays four at home, four 
on the road, and one bye. 

R5. First weekends. Each team must have home matches or byes at least on 
two of the first five weekends. 

R6. Rival matches. Every team except FSU has a traditional rival. The rival- 
pairs are Duke-UNC, Clem-GT, NCSt-Wake, and UMD-UVA. In the last 
slot, every team except FSU plays against its rival, unless it plays against 
FSU or has a bye. 

R7. Popular matches in February. The following pairings must occur at least 
once in slots 11 to 18: Wake-UNC, Wake-Duke, GT-UNC, and GT-Duke. 
R8. Opponent ordering constraints. No team plays in two consecutive slots 
away against UNG and Duke. No team plays in three consecutive slots 
against UNG, Duke and Wake (independent of home/ away). 

R9. Other idiosyncratic constraints. UNG plays its rival Duke in the last slot 
and in slot 11. UNG plays Glem in the second slot. Duke has a bye in slot 
16. Wake does not play home in slot 17. Wake has a bye the first slot. 
Glem, Duke, UMD and Wake do not play away in the last slot. Glem, 
FSU, GT and Wake do not play away in the first slot. Neither FSU nor 
NGSt have a bye in the last slot. UNG does not have a bye in the first 
slot. 

RIO. A small set of additions to the original description have been published 
very recently [141]: Every team must have an H in the first three slots. 
Every team must have an H in the last three slots. Wake is bye in the 
first slot and must end AH. 

Optimization criteria. There are several additional criteria the AGG re- 
quires of a time-table [115]. As Henz notes [69], some of these optimization 
criteria are conflicting, so the best one can hope for are Pareto-optimal solu- 
tions. The goal of the integer local search approach is to find solutions that 
are at least as good as the official 97/98 schedule. 

01. Avoid two opening aways. The number of teams that play away in the 
first two slots should be small. We denote this number by OAA. 

02. Good slots in February. Table 5.3 classifies (a) weekday and (b) weekend 
games into A-games, B-games and bad games (represented by 0) . If a slot 
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Table 5.3. Game quality 
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contains at least one A-game or at least two B-games, it is called an A- 
slot. If a slot is not an A-slot and contains at least one B-game, it is a 
B-slot. All other slots are bad slots. In February (slots 11 through 18), 
the A-slots should be maximized and the bad slots minimized. 

03. Home/Away/Bye pattern criteria. The number of occurrences of three 
subsequent home matches or byes should be small Similarly, the 

number of occurrences of three subsequent away matches or byes (AB 3 ). 
And again, for weekends the same criteria should hold {HB!^,, AB!^) . 



Any schedule can be rated according to the above optimization criteria, 
summarized by a vector of 7 numbers. The official timetable computed by 
Nemhauser and Trick meets these optimization criteria as follows: 

OAA HB 3 ABs HB!^ AB'^ bad A-slots 

i 4 3 5 4 2 3 



5.2.3 Integer Local Search Formulation 

In this section, a 0-1 integer linear local search model will be developed to 
state all the requirements. To simplify the model, we number the teams in the 
order given above. For the integer local search model, we introduce indices i 
and j that range over teams {1 ... 9} and t which ranges over slots {1 . . . 18}. 
The binary decision variables are Xijt and Xijt = 1 iff team i plays as guest 
of team j in slot t, for all 1 < i, j < 9. Additionally, a team index of 0 is used 
to express byes, e. g. Xiot = 1 iff team i is bye in slot t. Similarly, xm encodes 
homes, i. e. xm = 1 iff team i plays at home in slot t. 

We present the constraints from the most general ones to the more specific 
and finish with the idiosyncrasies of the ACC97/98 season. 
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RO. The following constraints implement the basic double round robin scheme, 
requirement RO. 

Every team plays at exactly one place (or is bye) in every slot. 

Xijt = 1, for all i > 0,t (5.1) 

0<j<9 

Every team is visited by at most one team in every slot. 

Xijt < 1, for all j > 0,t (5.2) 

0<i^j 

All pairings are consistent. 

Xijt Xjjt, for all 0 < i yf j and t (5.3) 

Double round robin: Every team plays every other team once away (and 
once at home which is implied in combination with (5.3)). 

Xijt = 1, for all i,0 < j ^ i (5.4) 

i 

R3. The following constraints restrict the allowed game sequences (Home/ 
Away/Bye pattern constraints), and need to be duplicated for the weekend 
slots. 

Treating bye as away, no more than 2 away games in a row. 

Xiis < 2, for all i, 1 < t < T — 2 (5.5) 

s=t...t+2 

Treating bye as home, no more than 3 home games in a row. 

{xiis + Xios) <3, for all i, 1 < t < T — 3 (5.6) 

Treating bye as away, no more than 2 home games in a row. 

Xiis < 2, for all i, 1 < t < T — 2 

s=t...t+2 



(5.7) 
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Formulating the Mirroring Scheme. 

Rl. To ease comparison with the previous approaches [115, 69], we will use 
the same mirroring scheme throughout our experiments. Due to requirement 
R9 which states that UNC and Duke meet in slots 11 and 18, perfect mirroring 
is not possible. For this reason, the N&T approach resorts to a mirroring 
scheme that switches slots 9 and 11 and obtain a mirroring scheme in which 
the minimal distance between any pair of teams is 7. The N&T approach 
also switches slots 8 and 9 (although no obvious constraint enforces this), 
and arrives at a set of mirrored slots of 

Mmt = {(1, 8), (2, 9), (3, 12), (4, 13), (5, 14), (6, 15), (7, 16), (10, 17), (11, 18)}. 

Note that fixing a conflict in a perfect mirroring scheme by switching entire 
slots also affects all the other pairings; all teams meeting in slot 1 will meet 
again in slot 9 — only because of the UNC-Duke meeting. We will discuss an 
alternative to this scheme in Section 5.2.7. The constraints that enforce the 
mirroring can be formulated as: 

Mirror return games. 

Xijs Xjit for all i yf j and (s,t) G M^t (5-8) 

Mirror byes. 

Xiot Xios for all i yf j and (s,t) G M^t (5-9) 

ACC Specific Requirements. 

R2, R4, R5. The following restrictions are more specific to the ACC and 
concern unliked sequences of H/A/B. 

No team finishes AA. 

Xi,i,T-i + Xj,o,T-i + Xt^i^T + Xifi^T > 1, for all i (5.10) 

Of the 9 Saturdays, each team plays four at home, four on the road, and 

one bye. 

Xiit = 4, ^ = 1 (5-11) 

i^2,4,...,T i^2,4,...,T 

Each team must be home or bye at least on two of the first five weekends. 

XiiH- Xjot > 2, for alH (5.12) 

i=1...5 

R6, R7, R8, R9, RIO. Finally, the team-specific requirements of the ACC97/ 
98. We do not present the constraints here as for most of these requirements 
the the constraint encoding is straightforward. An exception is constraint R6 
(rival games), for which an equivalent formulation can be used, i.e., three of 
the four rival games must be played in the last slot. 
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Optimization Criteria. The optimization criteria are modeled here as hard 
constraints to ensure that all resulting solutions will be at least as good as 
the official schedule. Soft constraints could be used alternatively, of course. 

01, 02, 03. Modeling 01 is straightforward: 

Avoid opening away/away (no more than 1 team). 

+ a;i,o,2) > 8 (5.13) 

i 

To formulate 02, additional variables need to be introduced for all slots 
in February; qtr = 1 if the quality of slot t G Feb is r, where r G {0,1,2}, 
r = 0 represents a bad slot, r = 1 a i?, and r = 2 an A slot, respectively (of 
course, '^^qtr = 1 for all t). An example from the constraints that link the 
X and q variables: 

Given the game quality matrix G™ for weekdays, every weekday slot in 
February that is marked as an A slot must contain at least one A or two 
B games. 



2 * 9 t 2 < ^ G“- * Xjit, for all weekdays t G Feb (5.14) 

Similarly for B slots and for week ends. 

Ensure the N/T quality level for game qualities. 

^ 9t2>3 ^ (7to<2 (5.15) 

ieFeb teFeh 

Similarly, to ensure the Home/Away/Bye sequences stated in 03, additional 
variables are introduced for every team and time point to that state when 
long unliked sequences occur starting at tp. For the full list of constraints, 
the reader is referred to Appendix A. 



5.2.4 Redundant Constraints 

There are certain simple truths about DDRR schedules that are implicit in 
any DDRR encoding and that can be explicated in order to improve the 
operational performance of integer local search. 

Such redundant constraints are often employed in other domain-indepen- 
dent frameworks to improve a representation with respect to the operational 
behavior of a solver when applied to it. In constraint programming, for exam- 
ple, redundant constraints can strengthen the propagation. In integer linear 
programming, redundant constraints are used to tighten the LP relaxation. 
Also, redundant constraints have recently been used for local search by Kautz 
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and Selman [94] in SAT planning models. The following redundant constraints 
are used in the DDRR model. ^ 

Every team is bye twice. 



^a;i 0 i = 2, for alH (5.16) 

i 

One team is bye in every time slot (holds for an odd number of teams 
only). 



^a;*ot = l, for alH 

i 

Half the teams are home in each slot. 

Xiit = 4, for all t 

i 



(5.17) 



(5.18) 



5.2.5 Previous (Multi-stage) Approaches 

The previous strategies to solve the ACC problem [115, 69] factor the prob- 
lem into several stages, following earlier approaches to sports scheduling by 
Cain [25], Schreuder [130], and others. Each stage of the problem is solved 
individually and ensures that a subset of the full set of constraints are met. 
Solving all stages in sequence yields complete timetables that meet all the 
constraints. 

(i) The first stage of the multi-stage approach generates so-called 
patterns: A feasible pattern is a sequence of H/A/B (one letter 
for each slot of the DDRR) which meets the particular mirroring 
scheme and all given H/A/B constraints. 

(ii) The second stage produces so-called pattern sets: a number of 
patterns (as many as there are teams) are selected from the col- 
lection of patterns into a pattern set. The chosen patterns of 
each pattern set must meet the condition that for every slot, 
there must be four patterns with an H, four with an A and one 
B. Additionally, one can require a pattern set to minimize the 
number of less preferred patterns [115] (e. g. requirement 01). 

(iii) The third stage, finally, renders the complete timetables. A 
timetable is computed on the basis of a given pattern set by 
assigning one team to each pattern in the set. A timetable is fea- 
sible if all problem constraints are met. In [115] ACC timetables 

^ We notice that there is an interesting connection to the pattern set approaches: 
The redundant constraints (5.17) and (5.18) express exactly the constraints on 
grouping patterns into pattern sets. 
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are computed with an additional intermediate stage that assigns 
team placeholders to pattern sets first. 

To solve the problem, the N&T approach [115] uses explicit enumeration in 
stage (i), integer programming in stage (ii), and integer programming and 
explicit enumeration in stage (iii). This indicates the difficulty of solving the 
problem. 

Henz [69] uses constraint programming to solve the individual stages. The 
CP approach, implemented in Oz [137], turns out to be significantly more 
efficient than integer programming with explicit enumeration, primarily be- 
cause the time-consuming final explicit enumeration phase is more efficiently 
accomplished by using constraint propagation.® 

5.2.6 Experimental Results under Varied Constrainedness 

In this section, we describe the experimental results of integer local search. All 
above constraints were modeled with the AMPL algebraic modeling language. 
After AMPL preprocessing, the constraints were handed to Wsat(oip) in 
expanded form using an AMPL-Wsat(oip) interface (Appendix A contains 
the full AMPL model). 

Table 5.4 shows a timetable found by Wsat(oip) for the entire set of 
constraints given above (RO-RIO, 01-03). The quality of the timetable im- 
proves the official timetable given in [115] with respect to several formalized 
quality measures (note that additional informal considerations actually lead 
to the selection of the official timetable [115]). 

OAA HBs ABs BB!^ AB'^ bad A-slots 

I 3 I 5 4 0 4 

The initial experiments of Wsat(oip) were carried out before all require- 
ments were available and lead to promising results.® As more and more con- 
straints unfolded, the solution times increased. To capture this behavior, we 
next present experimental results for a sequence of problems with increasing 
constrainedness. 

Table 5.5 investigates the scaling of Wsat(oip) with increasing con- 
strainedness. The problem instances start from general constraints of the 
DDRR scheme and successively incorporate more specific constraints. The 
number of solutions to the problem was computed using the Oz constraint 
program by Henz [69].^® We additionally ran an IP branch-and-bound pro- 
cedure (Cplex 5.0) on the given problems to obtain an estimate of its capa- 
bilities for the monolithic 0-1 ILP. 

® Oz is publically available from http:://www. ps.uni-sb.de/oz/. 

® I thank Michael Trick and George Nemhauser for sharing the requirements at an 
early stage. 

I thank Martin Henz for sharing the Oz program. Note that onr tightness level 
t5 corresponds to tightness 0 reported in [69]. 
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Table 5.4. A Schreuder-timetable computed by Wsat(oip) from an AMPL model 
of the constraints Rl-RlO, 01-03. The format is the same as the one used in [130]. 
-1-1 means home against 1,-1 means away at team 1, 0 means ’bye’. The particular 
run took 150s. 







slots 






1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 




Clem 1 


-48 -6 -44 -3 -2 -45 -9 -8 +6 -47 0 -4 -43 -42 -5 -49 -7 0 




Duke 2 


-5 -47 -49 -4 -41 -3 0 -45 -7 -48 -6 -9 -44 -1 +3 0 -8 -46 




FSU 3 


-46 -9 -7 -41 0 +2 -8 -6 -49 -5 -44 +7 -1 0 -2 -48 -45 -4 




GT 4 


-47 -8 -1 -42 -5 0 -46 -7 -48 -9 -3 -41 -2 -45 0 -6 -49 -43 




UMD 5 


-42 0 -6 -49 -44 -1 -47 -2 0 -43 -8 -46 -9 -4 -41 -7 -3 -48 




UNC 6 


-3 -41 -45 -8 -47 +9 -4 -43 -1 0 -42 -5 -48 -7 -9 -44 0 -2 


O! 

a 


NCSt 7 


-4 -2 -43 0 -6 -48 -5 -44 -42 -1 -49 -3 0 -46 -8 -45 -41 -9 


UVA 8 


-1 -44 0 -46 -9 -7 -43 -41 -4 -2 -45 0 -6 -49 -47 -3 -42 -5 




Wake 9 


0 -43 -2 -5 -48 -6 -41 0 -3 -44 -7 -42 -45 -8 -46 -1 -4 -47 



Table 5.5. Experimental results for increasing constrainedness. Columns report 
the constraints added, the ‘tightness’ level, the number of variables and con- 
straints of the problem, the total number of solutions (‘unkn’ means unknown). 
The Wsat(oip) columns report on runtimes with (4-red.) and without (—red.) 
redundant constraints (5.16)-(5.18). All runtimes are time to first solution on an 
Intel Pentium Pro 300Mhz (’-’ means no experiment performed, CNS=could not be 
solved in 12h runtime). For Wsat(oip), runtimes are averaged over 50 runs (4-red) 
20 runs (—red), and 10 runs^. The maximal standard error in column 4-red is 13%. 



constraints 

added 


cmnt 


lev 


n 


m 


#sols 


Wsat( 

—red. 


OIP) 

-4red.. 


Cplx 

5.0 


(5.1)-(5.4) 


DDRR 


to 








Is 


0.1s 


183s 


(5.5)- (5.7) 


H/A/B 


tl 


■Cgll 






15s 


0.4s 


649s 


Weekends 


H/A/B 


t2 


1620 


2520 


unkn 


79s 


Is 


873s 


(5.8)- (5.9) 


Mirror 


t3 


1620 


3249 


>le5 


27s 


9s 


174m 


(5.10)-(5.12) 


R2,4,5 


t4 


1620 


3285 


- 


631s 


162s 


CNS 


Idiosync. 


R6-R9 


t5 


1339 


3052 


321 


245mt 


1664s 


CNS 


(5.13) 


oi 


t6 


1335 


3047 


321 


- 


1484s 


CNS 


(5.14)-(5.15) 


02 


t7 


1359 


3069 


272 


- 


2128s 


CNS 


Opt. H/A/B 


03 


t8 


1773 


3466 


88 


- 


2847s 


CNS 


Recent add. 


RIO 


t9 


1773 


3479 


87 


- 


1798s 


CNS 



Discussion of Results. The approach by Nemhauser and Trick [115] 
yielded a complete set of solutions for the problem of tightness t9 with a 
turn-around time of 24 hours. The multi-stage constraint program [69] is cur- 
rently the most efficient approach and finds all solutions to the full problem, 
requiring only a few minutes (depending on the exact tightness of the prob- 
lem). Interestingly, opposed to local search, runtimes of the CP approach to 
find the first solution tend to decrease the tighter the problem is constrained. 
Conversely, it occasionally exhibits problems on loosely constrained variants 
of the problem, when the deterministic search happens to enter large subtrees 
that do not contain solutions. For example, tightness t4 does not render a 
solution within hours. When the number of generated patterns (first stage) 
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is large, the CP approach can also run into difficulties if useless pattern sets 
are produced first. 

Integer Local Search. The general observation from Table 5.5 is that as the 
the number of solutions decreases, runtimes of Wsat(oip) increase, as is to 
be expected with local search. What is surprising, however, is that solutions 
are still found when the constrainedness has reached a level that rules out all 
but 87 solutions! Another observation is that the redundant constraints play 
an important role for local search in this problem, in particular as the con- 
strainedness is increased (thus we did not conduct experiments above tight- 
ness level t5 without redundant constraints). More evidence is provided by 
the fact that posting additional constraints which do not change the number 
of solutions tends to decrease runtime. 

Throughout the experiments, Wsat(oip) was run in single-solution mode, 
although a multi-solution search would arguably be more appropriate for a 
comparison to the previous approaches. A multi-solution search would in- 
volve continuing search after having found a feasible solution, while avoiding 
previously generated timetables. Possibly, the search could be encouraged to 
remain ‘close’ to the previously found solutions. 

Comparison. Since local search is incomplete, it is not possible to search for 
the complete set of solutions and then stop. On the other hand, to be able 
to solve the problem, restrictive assumptions are made in the N&T and CP 
formulations (the mirroring scheme) which rule out a number of solutions to 
the problem that would otherwise be considered perfectly acceptable. The 
main advantage of the completeness of N&T and CP for this problem is thus 
their ability to detect when no solutions exist to the constraints. 

The main disadvantage of a pure local search approach is thus that there 
is no response if no feasible solution exists to a particular model. However, 
since the problem is formulated using integer constraints, LP relaxations can 
be employed: When tightening the constraints leads to IP infeasibility, the 
corresponding LP will sometimes also be infeasible. LP infeasibility can nor- 
mally be determined efficiently by linear programming. For example consider 
tightening constraint (5.5) to “treating bye as away, no more than 1 away 
game in a row”; the LP relaxation of the resulting model is proved infea- 
sible by Cplex in 75 seconds. Also, existing LP presolving can sometimes 
prove infeasibility. For instance, we accidentally swapped incorrect slots of 
the mirroring scheme and obtained an infeasibility warning from the AMPL 
presolver instantly. 

Parameters. Wsat(oip) was run with parameters t = 2, Pnoise = 0.01, Max- 
moves=2e6, Pzero = 0.9. Note that the runtime variation due to parameter 
variations turned out to be small (similar to the standard error for 20 runs). 
For example, changing parameters to Pnoise = 0.2, or additionally turning 
off history yielded similar results for these problems. However, a rigorous 
experimental analysis is beyond the scope of this monograph. 
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To obtain an estimate of the capabilities of IP branch-and-bound for the 
monolithic 0-1 ILP, we also ran Cplex on the given problems. All results 
report on standard parameter settings. Cplex was further tried with a fea- 
ture (‘sosscan’) that identifies special ordered sets (sos type 3, i.e. a set of 
binary variables that appear in a less-than or equality constraint with -|-1 
coefficients and an RHS value of -1-1), to apply special branching strategies. 
Despite 20% of the constraints being of this type, performance degraded with 
this option. We also attempted other option changes which did not improve 
performance, e. g. changing the branching direction (by setting the branch 
variable first to one in order to improve propagation), and “strong branch- 
ing” , recommended for hard pure integer programming problems. Note that 
we cannot rule out that a different IP model or other parameter settings 
might improve performance. 

5.2.7 Minimal Distortion Mirroring 

As mentioned earlier, in order to comply with the given team pairing con- 
straints that are in conflict with the mirroring scheme, the previous ap- 
proaches swap entire slots. Swapping slots, however, is problematic when 
several preassigned team pairings are in a conflict: For example, suppose 
that the team pairing {1,2} is required in slots ti,t 2 while pairing {3,4} is 
required in slots ti,t 3 (neither t 2 nor ts mirroring ti). In this case, if t 2 ^ 
swapping slots is not possible because two swaps would be required that 
are inconsistent with each other. To handle this case, we will resort to swap 
individual opponents instead of entire columns. 

We will refer to the slot mirroring some slot t in the original mirroring 
scheme as t' . Consider the case in which the input constraints fix the team 
pairing {a, 5} for slots ti < t 2 wherein yf t 2 (i-e. <2 does not mirror ti). 
Now instead of swapping column and t 2 like before, we limit the distortion 
of the original mirroring scheme. The idea is to pick two suitable teams c\ and 
C 2 (called sweeper teams) and make the following changes: Relax all mirror 
constraints involving a,b,c\,C 2 for the slots ti,t},t 2 ,t 2 add constraints 
to fix the pairings of the sweeper teams. Figure 5.1 illustrates the situation 
for the case of t\ and t 2 both being in the first half of the season. 

In this minimal distortion mirroring^^ , the basic mirror pattern is mostly 
preserved and additional team pairing requirements can be included. The 
distance to return games remains the distance of the perfect mirroring for 
all pairings except (i,j) € {a, 5, ci, 02 }^, t yf j, thereby improving the mean 
distance between a game and its return game. 

Modeling. In order to formulate minimal distortion mirroring, we employ 
additional variables to select the teams ci and C 2 . For each team 1 < t < 9, 
the binary variable j/i = 1 iff t is a sweeper team (of course, teams a and b 



I thank Martin Henz for coining this term in a discussion. 
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{a,b} 




{a. Cl} 




{a,C 2 } 


{C1,C2| 




{C1,C2| 




{b, C 2 } 




{b, Cl} 



Figure 5.1. Minimal distortion mirroring. 



cannot be sweeper teams). First, to achieve a minimal distortion, we require 
the number of sweeper teams to be minimal, i. e. yt = 2. 

Next, we need to reformulate the mirror constraints, choosing a perfect 
mirror Mp = {(s, s + 9) : 1 < s < 9}. While we mirror byes exactly as before 
(5.9), the following constraints substitute (5.8). 

Mirror return games as usual except in slots 

Xijs Xjit for all t yf j and (s,t) S Mp] s,t ^ 12 ,^ 2 } (5.19) 

In slots ti,t'i,t 2 , t' 2 , mirror return games as usual except for games against 

conflict teams (a and b) and the (current) sweeper teams. 

Vi V yj V (xys ^ Xjit) for all i yf j; i,j ^ a,b 

and {s,t) G Mp] {s,t} n {ti, t2, t' 2 } 7 ^ ® 

(5.20) 

In slot t\ and t 2 , if i and j are sweeper teams, they must meet. 

{yi A yj) {xijt V Xj^t) for all i < j] i,j ^ a,b] t G {h,t 2 } (5.21) 

In slot t'l and t' 2 , if i is a, sweeper team, it must meet a conflict team. 

y* ^ \/ {x^jt V Xjit) for all i^ a,b] t G {^ 1 ,^ 2 } (5.22) 

jG{a,b} 

Note that (5.21) and (5.22) are still linear inequalities. Given the above 
model of the ACC problem (R1-R9), we obtain the timetable in Table 5.2.7 
together with the automatically assigned sweeper teams from Wsat(oip) 
(average runtime was not measured in this experiment). 

In contrast to the Mnt mirror, which has an average distance between 
pairings of 8.11, the minimal distortion schedule achieves a distance of 8.80.^^ 
The quality vector of timetable 5.2.7 is: 

OAA HB 3 ABs BB'^ AB'^ bad A-slots 

0 4 3 5 4 I 3 



12 



Of the 9x9 = 81 return games (counting byes), only 8 have a distance of 7 from 
their hrst leg, all others have a distance of 9. 
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Table 5.6. A minimal distortion timetable from Wsat(oip), meeting the require- 
ments R1-R9. Preassigned games that conflict with the regular mirror are shown 
in boldface. To correct the conflict, two sweeper teams are selected, which happens 
automatically when the described model is solved. Games that do not follow the 
standard mirroring scheme are drawn in boxes (as well as the sweeper teams). 



slots 



1 2 3 

t'l 



4 5 6 7 



8 9 

f2 



10 



11 



12 13 14 15 16 17 



18 
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IClemI 

Duke 
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GT 

UMD 
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Wake 



-b2 

-1 bj +5 
-b8 El +7 
-h5 -8 +9 
-4 +9 -2 
-7 EH 0 
-h6 0 -3 

-3 -h4 -hi 
0 -5 -4 



0 +4 -7 -9 
-9 +7 -4 0 

+4 -9 -h6 -5 
-3 -1 +2 -6 
-7 -h6 0 -b3 

-b8 -5 -3 +4 
-b5 -2 +l -78 
-6 0 -79 -7 

-72 -73 -8 +1 



-75 ^ 
-78 El 

0 El 



-77 0 
-1 -8 
-9E3 

-4 +9 
-2 -75 
-76 -7 




-4 -77 -79 -5 Ea 
-7 -74 0 -8 f+T 

-79 -6 -75 0 El 

-71 -2 +6 -7 0 

-6 0 -3 -71 -78 

-75 -73 -4 -79 El 
-72 -1 -8 -74 -9 
0 -9 +7 -72 -5 
-3 -78 -1 -6 -77 



5.3 Conclusions 

This chapter has studied two hard timetabling/scheduling problems. The 
Progressive Party Problem and scheduling of the ACC97/98 basketball con- 
ference. Both problems can be formulated as 0-1 integer linear programs. It 
has reported the first ILP model for the ACC problem that we are aware of 
and improved the existing ILP model for progressive party. 

To the best of our knowledge, no previous techniques have been reported 
to solve either of the problems from a given 0-1 ILP representation (Smith et 
al. [136] report a number of unsuccessful attempts using integer programming 
branch-and-bound) . From the viewpoint of integer programming, the contri- 
bution of this chapter is thus to demonstrate that both problems can be solved 
in a 0-1 integer constraint encoding using a general solver (Wsat(oip)). 

From the applications viewpoint, we have shown in the first case study 
that the progressive party problem can be solved efficiently and robustly using 
Wsat(oip), i. e. the strategy scales gracefully with increasing constrainedness 
of the instances (the original study [136] investigated only one instance). 

The second case study has demonstrated that integer local search is able to 
find solutions to ACC97/98, a difficult and complex timetabling problem. We 
have shown that our approach yields solutions competitive with the official 
timetable (reported in [115]). We also presented minimal distortion mirroring, 
a mirroring scheme that can still handle the situation when team pairing 
requirements conflict with swapping time slots of a mirroring scheme, the 
strategy employed in previous (factorization) approaches. 

Additionally, the second case study in this chapter has studied the be- 
havior of Wsat(oip) across a range of increasingly tight problems. While we 
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have seen that the runtime of local search does increase on very tight prob- 
lems, we have demonstrated that even extremely tight problems can be solved 
by Wsat(oip). Moreover, we have shown that adding redundant constraints 
can help local search to find solutions more quickly. 




6. Covering and Assignment 



This chapter investigates two integer optimization problems, radar surveil- 
lance and course assignment. For both problems, the 0-1 OIP encoding is 
straightforward. Structurally, the problems are extensions of set covering and 
generalized assignment, respectively. The first problem stems from an indus- 
trial project at the Swedish Institute for Computer Science (SICS), while the 
second problem arose from an operating application at the Universitat des 
Saarlandes. Both studies in this chapter will focus on performance variation 
of integer local search and IP branch-and-bound with increasing problem size. 

Further, using the radar surveillance problems, we will perform experi- 
ments to determine the impact of the OIP representation on performance. 
The experiments demonstrate that the Wsat(oip) method critically depends 
on the soft constraint representation using constraint bounds. 



6.1 Radar Surveillance Covering 

The problem considered in this section is related to the classic NP-hard set- 
covering problem (see (SCP) in the Introduction). It extends set covering by 
complicating side-constraints that are specific to the radar domain and which 
prevent a direct application of domain-specific heuristics from the literature. 
On the other hand, its particular structure is well-suited for an encoding into 
OIP and makes it an intersting test case for integer local search. 



6.1.1 Problem Description and Formulation 

The case study and its basic modeling originate from a project currently 
carried out at the Swedish Institute of Computer Science (SICS) [24]. The 
goal is to plan radar surveillance of a geographic area. As customary in the 
radar surveillance domain, the area is divided into hexagonal cells. As part 
of the problem statement, a number of radar stations are given that are 
located in fixed cells on the map. The problem is to find a static plan that 
determines for every cell c by which radar stations c is observed, subject to 
the constraints that each cell be observed by at least three stations. Figure 

6.1 gives an illustration. 
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Each radar station can divide its signal scope circle into six sectors and can 
vary the signal strength in each sector independently from zero to some given 
maximum distance dmax- Aside from some insignificant cells, the majority 
of the cells must be covered by 3 radar stations (desired coverage 3) and all 
coverage beyond this is to be minimized (over-coverage) . Small over-coverage 
is desired for economic reasons as well as for reasons of detectability, i. e. radar 
can more easily be detected in areas with a high exposure. 

Because of the placement of stations, some cells cannot (physically) be 
covered by at least three stations and hence must be covered by as many 
stations as possible (and can then be factored out from the problem). In the 
original model, a radar station can be switched on to cover only the cell that 
it is located in. It always covers it provided it is switched on for some sector. 



sector 2 




Figure 6.1. Radar map with hexagonal cells. 



The problem can be modeled by the following over-constrained 0-1 integer 
program. For every combination of radar unit u, sector 1 < s < 6 and possible 
observation distance 1 < d < dmax, a Boolean variable ausd is introduced. 
Variable ausd = 1 if and only if station u is switched on in sector s at distance 
d. The set of all cells that station u reaches in sector s at distance d is denoted 
by Cusd- The over-constrained integer program (OIP) model is as follows. 

Cover each cell. There are significant and insignificant cells. While in- 
significant cells should not be observed, significant cells must be covered 
by at least three stations. 

ausd > Dc, for all c, (6.1) 

C e Cusd 

where for all cells c, Dc = 3 if c is significant and Dc = 0 if c is insignif- 
icant. For any c, ausd leads to all stations u that can reach c (and s, d 
yields their respective observation field in terms of sector and distance 
where c is reached) . 
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Consistency. If station u is switched on at distance d > 1 in sector s, it is 
also be switched on at distance d — 1 in sector s. 

<Jusd - cTusd’ < 0 for all u, s, 1 < d < dmax, d' = d-l. (6.2) 

Soft Constraints. Minimize over-coverage. Cells should not be exposed 
beyond their desired coverage. 

{soft) ^ Uusd < Dc, for all c. (6.3) 

^ ^ usd 

It is important to note that in order to minimize the total over-coverage, 
minimizing the number of violated soft constraints is not sufficient. Over- 
coverage can occur in different degrees for each cell. 

Minimizing Over-coverage in the OIP. To understand the OIP mini- 
mization problem, we first observe that it is confined: for every soft constraint 
cx < d, there exists a hard constraint cx > d. Therefore, the OIP minimiza- 
tion problem has the linear objective function 

min E( E ^usd) -^cj (^■^) 

^ C G Cusd 

and can directly be translated to an integer linear program using (6.4). The 
ILP can be approached with IP branch-and-bound and the linear relaxation 
can be used to compute lower bounds on the over-coverage. 

Relation to Set- Covering. As observed above, the radar surveillance prob- 
lem shares some of its structure with the set covering problem (SOP). It is the 
side constraints of physical consistency (6.2) that avoid the direct application 
of domain-specific methods for set-covering. Also, it should be noted that we 
are not currently aware of an AP-hardness result for the radar-surveillance 
domain. 



6.1.2 Experimental Results under Varied Problem Size 

This section reports on experimental results for a collection of radar surveil- 
lance instances that were generated according to different characteristics. All 
instances were randomly generated and vary in size (100 to 2100 cells), in the 
percentage of insignificant cells (0%, 2% and 5%), and in the spread of radar 
stations on the map (even or uneven). The density of stations remained con- 
stant. Since real placement information for radar stations was not available, 
insignificant cells were randomly positioned on the map. Table 6.1 summa- 
rizes the experimental results, based on a suite of radar instances generated at 
SICS. We ran integer local search Wsat(oip), IP branch-and-bound Cplex 
5.0, a 0-1 simulated annealing strategy GPSiMAN [33], and the Cplex 5.0 
linear programming optimizer. 
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Parameters. Wsat(oip) was run with standard parameters {phard = 0-8, 
Pzero = 0.5,Pnoj5e = 0.01, t = 1) and with varying settings of Maxmoves for 
the different problem sizes: 30K, lOOK, 300K, 500K respectively. Cplex was 
run with different variations of the standard/auto parameter settings. We 
only report standard parameters since this yielded the overall best perfor- 
mance for the collection of instances. Cplex was limited to 12h of computa- 
tion time and did not reach the given memory limit of 400MB. For GPSIMAN, 
we used the following parameters: Maximum neighborhood size suggested by 
the solver (372, 776, 4447, 10772, respectively), 20 runs, 100 iterations, high- 
est level of reoptimization, default cooling schedule (including re-heating). 

Experiments with Constraint Programming. Various models (finite domain 
integer and Boolean) and enumeration schemes have been tried [24]. Al- 
though small problems are solved to optimality quickly, the larger sample 
instances could be solved with large over-coverage values only, given reason- 
able time. We hypothesize that it is thrashing that makes these problems 
hard for a constraint program that backtracks chronologically: Two distant 
radar stations hardly affect each other, yet with chronological backtracking 
the state of one station is only changed after visiting the complete subspace 
of configurations of many other stations. 

Discussion. According to Haridi et al., the long-term goal of the project is to 
cover a large geographical area with thousands of cells. It is thus an important 
criterion of success that the solution strategy scale well. The experimental 
results clearly show that while both IP branch-and-bound and GPSIMAN can 
handle small problems efficiently, problems of realistic size are beyond their 
size limitations. 

In contrast, Wsat(oip) is very effective on the sample problems of this 
domain, even for realistically sized problems. Only one class of problems 
with many insignificant cells (essentially ‘holes’ in the map) was difficult 
to solve. We did not systematically make attempts to improve performance 
on these instances because it is unknown if realistic maps would show this 
characteristic.^ 

From the LP optimal solutions, one observes that for many instances of 
the sample, the LP relaxation is tight, i.e. the optimal value of the over- 
coverage is the same as the LP lower bound. ^ Such problems are usually 
easier than problems with larger relaxation gaps (under otherwise similar 
parameters). The difficulty for the IP is thus closely linked with the size of 
the problems. Interestingly, in many cases the LP relaxation optimization 
takes longer than computing the optimal IP solutions using Wsat(oip). 

Dropping the Constraint Bounds. The surprising effectiveness of integer 
local search in this domain raises the question what the reasons are for the 



^ Increasing noise and decreasing p^ard provides some better solutions. 
^ We thank Alexander Bockmayr for initially pointing this out. 




Table 6.1. Experimental comparison for radar surveillance problems: Columns are problem size in number of cells and stations 
(stations have a maximal reach of dmax = 4), encoding size in number of variables n and clauses m, spread of stations on the map, 
percentage of significant cells, and LP lower bound for over-coverage, oc* gives optimal over-coverage (integer). Cplex 5.0 columns are: 
best over-coverage found, time-to-best solution, and total runtime. Cplex was run with standard/auto parameter settings. GPSIMAN: 
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Table 6.2. Performance drop of Wsat(oip) when dropping constraint-bounds. 
STD repeats the optimal solutions from the previous table, ‘no-bounds’ reports on 
the best solution found in 20 runs, and total reports total runtime. 



size 


spread 


%sig 


oc* 


STD 


Wsat(oip) 

no-bounds 


total 


100:22 


even 


100 


0 


opt 


opt 


16s 


200:44 


even 


100 


1 


opt 


8 


48s 


900:200 


even 


100 


2 


opt 


47 


314s 


2100:467 


even 


100 


3 


opt 


93 


1144s 


100:22 


even 


98 


1 


opt 


3 


14s 


200:44 


even 


98 


2 


opt 


9 


58s 


900:200 


even 


98 


4 


opt 


56 


448s 


2100:467 


even 


98 


12 


13 


166 


866s 


100:22 


uneven 


100 


3 


opt 


5 


16s 


200:44 


uneven 


100 


0 


opt 


1 


58s 


900:200 


uneven 


100 


5 


opt 


36 


564s 


2100:467 


uneven 


100 


9 


10 


102 


1132s 


100:22 


even 


95 


4 


opt 


10 


16s 


200:44 


even 


95 


5 


opt 


12 


58s 


900:200 


even 


95 


19 


25 


95 


290s 


2100:467 


even 


95 


- 


96 


266 


756s 



performance. To address this question, we performed the following experi- 
ment, which is based on the hypothesis that the performance is related to 
the OIP problem structure. Each problem instance was modified by changing 
the bounds of the soft constraints from ‘< 3’ to ‘< O’. From Proposition 3.1.4 
(in Section 3.1.3), we know that tightening bounds of a confined OIP does 
not change the set of solutions. Further, we can account for the shift in the 
objective function by subtracting a value from the resulting objective, 3ng (if 
Us is the number of insignificant cells). 

Table 6.2 reports on the experimental results. Parameters were manually 
re-tuned to adjust for the change of the representation, resulting in switch- 
ing off both the tabu mechanism and history-based tie breaking. The results 
in the table demonstrate that the constraint bounds are critical to obtain 
the previous performance. This result is consistent with our expectation be- 
cause dropping the constraint bounds effectively makes the repair strategy 
blind with respect to which soft constraints are violated. When dropping the 
bounds, the search thus looses its focus and blindly makes perturbations of 
the variable values. 
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6.2 Course Assignment 

The course assignment problem considered in this section deals with assigning 
students into pre-planned courses according to their preferences. The study 
was carried out based on real data of the School of Law of the University 
des Saarlandes in the semesters of Summer 97 and Winter 97/98, and the 
obtained results were used by the school to assign students to classes. As the 
semesters have different numbers of students, the task created a collection of 
real problems of varying size, ready-to-use for an investigation on real data. 
The problem under consideration is related to the generalized assignment 
problem (GAP) but includes additional side constraints. 



6.2.1 Problem Description and Formulation 

The scenario is the following. Students of a law school (up to 500 per semester) 
have to be assigned to courses (up to 30) with pre-assigned time slots and 
rooms. The law school offers to use a Web based interface^ to register for 
a number of legal fields according the students’ current interests. Further, 
students may submit a timetable stating the preferred time slots and the 
slots which they are unable to attend {disliked slots or aversions). As several 
courses are taught in each field, the aim is to assign students to courses 
maximizing the overall satisfaction (satisfying aversions and preferences) such 
that every student is assigned to one course in every one of her registered 
fields, while the capacity of the courses is not exceeded and the courses are 
not filled too sparsely. 

The formulation of the problem is stated in the following. Table 6.3 sum- 
marizes the indices, constants and sets. The set F denotes the different legal 
fields, each field f G F is represented by a set Cf of courses. The set C con- 
tains all courses and and Ck is the desired number of participants of course 
k € C (usually the average number of participants of a course within the 
field). The fields for which a student i is registered are given by Ri C F. 
Further, the student preferences are part of the problem statement and are 
encoded by binary constants, pik = 1 if student i prefers the slot of course k 
and otherwise 0. Conversely, Uik = 1 if i has dislikes course k (because k is 
taught during a time slot which i cannot attend) . 

The aim is to fill the courses within given upper and lower limits while 
minimizing the number of disliked assignments and minimizing the number of 
unsatisfied preferences (in this order) . The problem can be encoded as follows: 
For every student i and course k, use a variable Xik = 1 if i is assigned to fc, 
otherwise 0 (if i is not registered for the field of course k, Xik is 0). 

OIP Formulation. The OIP formulation uses the following constraints. 



® An application provided to the students at Saarbriicken by Reinhard Schu. 
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Table 6.3. Parameters for the course assignment problem. 



Index 


Definition 


i 


Index for students. 


kj 


Indices for courses. 


f 


Index for field. 


Symbol 


Definition 


F 


Set of legal fields. 


C 


Set of all courses. 


R^ 


Fields which student i is registered for. 


Cf 


Set of courses in field /. 


C-k 


Desired number of participants of course k. 


Pik 


binary constant, 1 iff f prefers k. 


O^ik 


binary constant, 1 iff f dislikes k. 


m^ 


Upper bound on satisfiable preferences for i. 


Ucap 7 leap 


Relative upper/lower capacity limits for courses. 



Every student must attend exactly one course of each field she is registered 
for. 



Xik = 1, for all students i, and fields / S Ri. (6.5) 

keCf 

The number of participants of a course may not exceed the desired number 
of participants by more than Ucap and not fall below leap- 

leap ■ Cfc < E Xik < Ucap • Cfc, for all courses k G C. (6.6) 

i 

No student can visit two courses that temporally overlap. 

Xik + Xu < 1, for all students i, registered fields s < t, 

(6.7) 

such that k G Csjl G Ctj und fc, I overlap. 

Soft Constraints. For every student, minimize the number of aversions, 
i.e. the number of assigned courses that are disliked due to their time 
slot. 



{soft) Qik • Xik < 0, for all students i. (6.8) 

l<fc<s 

Soft Constraints. For every student, minimize the number of unsatisfied 
preferences. 



Pik ■ Xik > krii, for all students i. 

Kk<s 



{soft) 



(6.9) 
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where rrii is an upper bound on the number of satisfiable preferences: 
mi = min{|i?i|,^^-Pij} for student i where |i?i| is the number of regis- 
tered fields and py is the number of preferred courses of i. 

The reasoning behind the bound is as follows: Obviously, the number of 
registered fields is an upper bound on the number of satisfiable prefer- 
ences. However, if a student has less preferred courses than registered 
fields, the number of satisfiable preferences is the maximum number of 
non-overlapping preferred courses that cover all registered fields. To keep 
the modeling simple, however, we approximate this value by the number 
of courses in preferred time slots which is a valid upper bound. 

In order to account for the order of the goals (minimize the number of 
aversions first), constraints (6.8) are weighted such that constraints (6.9) are 
always dominated. 

ILP Reducibility. In order to apply lower bounding from the direct ILP 
conversion, we establish confinedness first. The given OIP (6.5)-(6.8) is con- 
fined. 

Proof: The soft constraints (6.8) are confined as all coefficients Oik are pos- 
itive. The soft constraints (6.9) are confined because the upper bounds are 
valid according to the above reasoning. □ 

Given the confinedness of the problem, we can directly reduce it to an ILP 
without the need to introduce additional variables. All IP branch-and-bound 
experiments are subsequently applied to the transformed OIPs. 

Note that in the given OIP minimization problem, the objective function 
value is given as a pair A-P, where A is the number of aversions in the 
assignment and P relates to the number of unsatisfied preferences. P may 
overestimate the number of unsatisfied preferences because is an (approxi- 

mate) upper bound on the exact number of satisfiable preferences for student 
i. 



6.2.2 Experimental Results under Varied Problem Size 

Table 6.4 reports on results of Cplex 5.0 and Wsat(oip). Both solvers 
were run with standard parameter settings. Additionally reported are Cplex 
results using strong branching. Several other parameter settings have been 
tried (e. g. different root heuristics) but did not improve performance. Cplex 
tree memory was bounded to 400 megabytes to avoid paging, runs marked ^ 
were cut-off to avoid paging and did not prove optimality. 

Relation to the Generalized Assignment Problem. The standard gen- 
erlized assignment problem can be formulated as follows. Let / be a set of 
agents and J be a set of jobs. For i G /, j G J, define Cy as the cost of assign- 
ing job j to agent i, Vij as the resource required by agent i to perform job j, 
and bi as the capacity of agent i. Let Xij be the binary decision variable that 
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is 1 if agent i performs job j and 0 otherwise. The Generalized Assignment 
Problem (GAP) is 



minimize 






i&i jeJ 


subject to 


1 ) j ^ 




iGl 




^ ^ ^ij ^ij bi^ f C 








Xij G {0, 1} 



(GAP) 



The course assignment problem differs from the standard GAP in the no- 
overlap constraints (6.7) and in the capacity constraints (6.6) that enforce a 
lower level of participation for every course. 

Related Work on the GAP. Gattrysse and VanWassenhove [27] report that 
most existing techniques for the GAP are based on branch-and-bound with 
bounds supplied through heuristics and through relaxations of the original 
problem (not necessarily linear programming relaxations). According to [27], 
bounds are usually derived from relaxation of the assignment or capacity 
constraints and a variety of techniques have been applied to the GAP. Some 
of the existing techniques might thus be applied to the course assignment 
problems. 



6.2.3 A Related Application: Reviewer Assignment 

A domain very similar to course assignment is the assignment of reviewers in 
a scientific conference. For GP97^, this method was employed to assign re- 
viewers to conference submissions. In a bidding, all PG members stated their 
paper preferences. The information provided by this bidding was sufficient 
to assign 90% of the papers automatically. The papers with an insufficient 
number of bids were manually assigned by the program chair, according to 
the PG members’ profiles and load of assigned papers. The method worked 
surprisingly well: In the final assignment, the load of all PG members was 
evenly distributed and most preferences could be satisfied. 

The problem was modeled using pseudo-Boolean constraints and solved 
to optimality by integer local search and integer programming branch-and- 
bound independently. 

Problem Formulation. In a bidding, all PG members submit a preference 
list of papers to review. Each PG member is allowed a maximum of 30 review 



The third International Conference on Principles and Practice of Constraint 
Programming. This section reports on joint work with Cert Smolka, program 
chair of CP97. 
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preferences in three classes: (3) major interest, (2) interest, or (1) minor in- 
terest. The goal is to assign as many reviewers to as many papers as possible 
(not more than three for each paper) while maximizing the reviewers’ satis- 
faction and assigning a similar number of papers to all PC members. In the 
following, we describe a modeling of the problem using an over-constrained 
integer program. 

The representation uses Boolean decision variables Oij = 1 iff reviewer i 
reviews submission j. The biddings of the PC members are represented by the 
sets Bl,B^,Bf according to the preferences of PC member i (in increasing 
interest). 

The constraints can be described informally as follows. Save resources: 
The number of automatically assigned reviewers for any paper is at most 3. 
Fair load balancing: No reviewer is to be assigned more than u papers or less 
than I papers (unless the bidding contains less preferences). Maximize the 
number of reviewers assigned to each paper. Maximize the number of class-3 
paper assignments for every reviewer. And finally, minimize the number of 
assigned class- 1 papers for every reviewer. 

In the review process, the main goal is to distribute all papers. Thus the 
soft constraints to maximize the number of reviewers for each paper were 
strictly preferred over the other soft constraints by assigning them appropri- 
ate (high) weights. This enforces that papers will be distributed three times 
if possible without violating the hard load balancing constraints. 

Results. CP97 had 132 paper submissions and thus a load of 396 reviews 
were to be assigned to 26 PC members. We applied three methods to solve the 
over-constraint pseudo-Boolean system: Wsat(oip); a linear programming 
package, Loqo [145] that provided sharp lower bounds in all experiments; 
and a public domain integer solver, LPsolve, by Michel Berkelaar. For opti- 
mization, the solution quality of an assignment was computed as the summed 
net violation of all (weighted) soft constraints. Modeling the constraints was 
done from an Oz program. For the linear programming based methods, the 
soft constraints were translated to an equivalent modeling using an objective 
function. For all programs, the runtimes were under 2 seconds. 

The assignment was constrained to be balanced: No PC member was 
to review more than u = 20 or less than I = 14 papers. The automatic 
assignment by this methods was well-received. On average, 86% of the class- 
3 preferences could be satisfied, and 32% of the class-2 preferences. Taken 
together with the papers with insufficient bids (that were manually assigned 
by the program chair) 41% of the class-1 preferences were satisfied. 
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6.3 Conclusions 

In this chapter, two 0-1 integer optimization problems, radar surveillance 
covering and course assignment, have been studied, whose difficulty to a large 
extent is a result of their unavoidable size. For each of the problems, a confined 
OIP encoding was given, which was directly converted to a corresponding ILP. 

Experimental results of integer local search (Wsat(oip)) and IP branch- 
and-bound (Cplex) have been reported for both domains. While for both 
domains, similar results were obtained for small problem instances with both 
frameworks, the experiments have shown that their scaling properties differ 
largely. Because integer local search exhibits a much more graceful scaling, 
Wsat(oip) was able to outperform Cplex by orders magnitude (runtime) 
on some of the largest given problems. 




7. Capacitated Production Planning 



“An important and widespread area of applications concerns the man- 
agement and efficient use of scarce resources to increase productivity.” 



[Nemhauser and Wolsey, 1988] 



Production planning is an important task in manufacturing systems and 
gives rise to a variety of optimization problems. Here we study a real-world 
lot-sizing problem from the process industry (manufacturing of chemicals, 
food, plastics, etc.). The problem is expressed as follows: given a set of prod- 
ucts and a collection of customer orders with due dates, construct a minimal- 
cost production plan such that all orders are met in time without exceeding 
resource capacity. The total cost of a plan consists of inventory and labor 
costs. 

The problem under consideration is similar to the well-studied capacitated 
lot-sizing problem (CLSP, see [43] for a survey) but includes the requirement 
of discrete lot-sizes that prevents a direct application of domain-specific meth- 
ods from the literature [41, 95, 72]. We therefore approach the problem with a 
new domain-independent heuristic for integer optimization, Wsat(oip), and 
empirically compare it to a commercial mixed integer programming (MIP) 
branch-and-bound solver (Cplex 5.0). 

This chapter describes a case study of Wsat(oip) on a large CLSP with 
discrete lot-sizes and fixed charges. We compare the experimental results on 
real data to Cplex applied to a tight integer programming model. We find 
that MIP branch-and-bound can only solve a sub-class of the CLSP with 
discrete lot-sizes, namely the problem where fixed charges and lot-sizes are 
equal. Further, Wsat(oip) is considerably more robust than Cplex in find- 
ing feasible solutions in limited time, in particular as the capacity constraints 
are tightened. With respect to production cost, both methods find solutions 
of similar quality. We examine fixed-capacity and varied-capacity problems. 
Using a Lagrangean relaxation technique we provide lower bounds that prove 
that the fixed-capacity problems are solved with near-optimal overall cost. 
We show that substantial savings can be achieved by varying capacity. 
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Table 7.1. Parameters for the CLSP with discrete lot-sizes and fixed charges 
(economic production quantities, EPQs). 



Index 


Definition 


i 


Index for items/products. 


t 


Index for time periods. 


Symbol 


Definition 


L. 


Lot-size of product i. 


E, 


Economic production quantity of product i. 


Dit 


Demand of product i in time period t. 


Tt 


Total labor units available in time period t. 




Unit labor requirement for product i. 


a 


Cost of carrying product i per unit/period. 




Future demand of product i starting period t. 


T 


Number of periods. 


N 


Number of items. 


S 


Cost per labor shift. 



7.1 Capacitated Lot-Sizing 

The problem under consideration can be classified as single-level, dynamic- 
demand capacitated lot-sizing problem (CLSP) with discrete lot-sizes and 
fixed charges. Given is a set of products and a number of customer orders (or 
forecasted demands) with due dates on a finite planning horizon. The goal 
is to compute a minimal-cost production plan such that all customer orders 
are met in time. No lateness or shortage of orders is permitted. Products (or 
items) can be produced in discrete periods of the planning horizon (weeks). 

Because production consumes resources and resources have limited ca- 
pacity, items often have to be produced earlier than needed and carried to 
the period where they are shipped. Such carrying incurs inventory cost (op- 
portunity cost of capital and storage cost) which is one of two cost factors 
in the problem considered here. Solving the CLSP optimally is known to be 
NP-hard [19]. Table 7.1 specifies the problem parameters. 

The CLSP considered here has two particularities: (i) Items can only be 
produced in predefined quantities (lots) and setup costs are compensated by 
economic production quantities (EPQs). At any time, production of item i 
is possible in quantities of 0 or -|- fc • Li, where k > 0, Li is the lot-size 
and Ei is the EPQ for item i (every EPQ is a multiple of the lot-size), (ii) 
The only resource is labor, available in either one or two shifts in any period. 
The amount of available labor has an associated cost (labor availability and 
consumption are expressed in cost units). Thus, production cost is equal to 
the sum of labor and inventory costs. 

In the problem, labor capacity can be varied between one and two shifts. 
Because less capacity enforces earlier production of items, a tradeoff exists 
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between labor and inventory costs. Because labor costs dominate inventory 
costs, reducing labor is critical to substantially save costs. However, due to 
practical considerations it is not acceptable to have too many labor level 
changes; thus the number of labor level changes considered was limited to 2 
in our experiments. 

To optimize the overall problem, we take the approach to solve a series of 
capacitated lot-sizing problems with different ‘labor profiles’ and choose the 
best solution, as follows. 

Labor Profiles. Labor consumption varies between items and is expressed 
by parameters Ri in terms of resource consumption per production of one 
unit of item i. In any period t, the total labor consumption is limited by 
Tt, available in one or two shifts. One shift incurs a per-week cost of S, 
two shifts incur 2S. A labor profile thus corresponds to a set {(t, 7t) | 1 < 
t < T,Tt e {S', 25'}}. Possible labor profiles are restricted to the pattern 
2-shifts/ l-shift/2-shifts as shown in Figure 7.1 and can be denoted by an 
interval [si, S 2 ] referring to periods si . . . S 2 on one shift, and periods 1 . . . si — 
1 and S 2 + 1 . . .T on two shifts. The cost of a labor profile [si,S 2 ] is thus 
(T — (s2 — Si -l- 1)) • 25 -l- (s2 — Si -l- 1) • 5. 



2shifts 



1 shift 



1 



I 

i 



j 

!si 



s2 



52 



week 



Figure 7.1. Valid labor profiles. 



Every labor profile has an optimal inventory cost. If labor could be freely 
varied, the labor availability would have to be modeled with problem vari- 
ables. However, since the number of allowed labor profiles is small, we factored 
the labor variability out from the optimization problem and approached the 
problem by solving each permitted labor profile, optimizing one CLSP at a 
time. Possible shift boundaries [si,S 2 ] were generated starting with si = 1 
and an initial one-shift period length I (s 2 = si -I- / — 1). Iteratively, S 2 was 
then increased as long as Wsat(oip) found feasible solutions for the result- 
ing CLSP (for Cplex, as long as infeasibility was not proved). If no feasible 
solution was found (for Cplex, if infeasibility of the profile was proved), si 
was increased to the next period and S 2 was reset. The two different integer 
solvers require different algebraic models which are described in the following. 
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7.2 Integer Local Search Formulation 

The integer local search model is a straightforward OIP. Production quan- 
tities per item and time period are expressed by finite domain variables pu 
that range over the allowed production quantities (and are bounded by the 
summed future demand flit). 

Pit & {p < \p=0\/p = Ei + k-L} 

where fc = 0, 1, 2, . . . , for every item i and time period t and flu is determined 
as flit — ^^t<s<T 

To formulate the constraints, we will make use of the abbreviation 
representing the amount of product i carried in inventory in time period t 
(textually substituted in the constraints): 

i 

5'[z, t] — ^ ( Pis Eis 
s=l 



The OIP formulation is as follows. 



^[z, t] > 0 


Vz, t 


(NOH) 


Rz ■ Pit < Tt 


Vt 


(CAP) 


(soft) Ci ■ S[i,t] < 0 




(INV) 



Negative-on-hand constraints (NOH) ensure that all orders are met in time. 
Capacity constraints (CAP) express that available labor capacity is not to 
be exceeded. The soft constraints (INV) express the competing objectives of 
minimizing inventory costs; for every item and time period, the inventory 
cost from carrying material has to be minimized. For every feasible solution, 
the resulting objective (the total inventory cost) is the summed violation of 
all soft constraints measured by ||.|| (OIP). Notice that for every soft (INV) 
constraint, there is a corresponding (NOH) constraint, thus the OIP is con- 
fined. Using finite domain variables to model production, the local search 
progresses by moving production up or down in allowed quantities induced 
by the violated constraints. 

0-1 Integer Model. The first modeling attempt used an over-constrained 
0-1 integer model with a logarithmic encoding of production quantities 
{EiXi + LiX 2 + 1LiXz + 4:LiX4 -|- . . . ). In addition to the blowup of the number 
of variables for this model, running Wsat(oip) did not yield solutions of ac- 
ceptable quality. We put this failure down to the fact that with a logarithmic 
encoding, a small change of production often requires a long sequence of local 
moves. For example, an increase from 2^ — 1 to 2^ lots can only be achieved 
by flipping fc -I- 1 variables. This appeared to be a strong hindrance of the 
search process. 
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7.3 Mixed Integer Programming Formulation 



This section requires some familiarity with integer programming terminology, 
as covered for example in [114].^ The sets and variables defined in the mixed 
integer programming model (MILP) are given in tables 7.1 and 7.2. The 
problem formulation (P) is as follows. 



P : 



N T 



minimize 






(7.1) 



subject to 





= 


> 

+ 


(7.2) 


Xtt 


= 


Uyu yi&SKUi 


(7.3) 


Xtt 


= 


EiZit + LiUit Vi G SKU 2 


(7.4) 


^i^it 


< 


Xit ^ Vz G SKU 2 


(7.5) 


t 

^ ^ Xik 


> 


t 

L,\J2D,k/L,-] \fieSKUi,t 


(7.6) 


k^l 




k^l 




t-1 




t-1 t 




^ ^ Xik 


> 


^ ^ DikZit + ^ ^ Dik{\ Zit) 




k^l 




k^l k^l 








\Ii G SKU2,t 


(7.7) 


^ ^ RiXit 


< 


Tt yt 


(7.8) 



Zit & {0,1}, y It integer 



In the MILP model, equation (7.1) represents the sum of total inventory 
carrying costs. Equation (7.2) is the material balance in each time period 
and equations (7.3)-(7.4) determine the total production quantity of each 
product in time period t. Note that binary variables are only defined for 
i G SKU 2 - Equation (7.5) states that if Zu is non-zero, then the minimum 
amount (EPQ) must be produced, and cannot exceed the bound f2u (only 
for items in SKV 2 )- 

Equations (7.6)-(7.7) represent constraints that tighten the relaxation gap 
between the integer solution and the LP relaxation of the problem. Equation 
(7.7) states that if product i is produced in period t, then the total amount 
produced up to period t — 1 must meet the total demand up to period t — 1. 
However, if the product is not made in period t, then the amount produced 
up to period t — 1 must meet the demand up to period t. From our observa- 
tion, this equation reduces the relaxation gap significantly and helps reduce 
the number of nodes branched on in a branch-and-bound solution method. 

^ The MILP modeling and Cplex experiments were carried out by Ramesh Iyer 
and Narayan Venkatasubramanyan. 
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Table 7.2. Sets and decision variables for the MILP model. 

Sets 

SKU Set of products (stock keeping units). 

SKUi Set of products for which lot-size (Li) is equal to eco- 
nomic production quantity (Ei). 

SKU 2 Set of products for which lot-size is a multiple of eco- 
nomic production quantity. 

Variables 

Sit Amount of product i carried in inventory in time period 
t. 

Xit Amount of product i produced in time period t. 

Uit Number of lots of product i produced in time period t. 

Zit Binary variable which is one if product i is produced in 

time period t. 



Finally, equation (7.8) represents the labor constraints that link the problems 
across all products. 

Due to the modeling of discontinuous integer values {xa G {0,Ei,Ei + 
Li .. .}) for items i G SKU 2 with binary variables zu, solving large problems is 
extremely expensive. We therefore attempted a Lagrangean relaxation tech- 
nique (see [17] for an overview of Lagrangean relaxation) where the problem 
is decomposed by relaxing the equations (7.8) to obtain the value of binary 
variables and then solving problem (P) for fixed value of binary variables, 
thereby solving subproblems that are less expensive to solve in each step. 

7.3.1 Lagrangean Relaxation Approach 

The Lagrangean relaxation method used for solving the problem (P) relaxes 
the complicating constraints (7.8) using Lagrange multipliers, thus resulting 
in a relaxed problem that is decomposable for each i. The relaxed problem 
(PL) is as follows 



NT T N 

PL:minimize [^ ^ CiSit] - ^ At '^{RiXu - Tt), 

it it it 

2=1 

subject to Equations (7.2)-(7.7). 

Thus, (PL) is a relaxation of (P) and represents a lower bound to the 
solution of (P). Since (PL) is decomposable with respect to i, each subproblem 
is combinatorially less complex, and can be solved to determine the variables 
Zit- Then, for fixed values of zu, the problem (P) may be solved to determine 
a specific solution that is an upper bound to the solution of (P). 

We note that due to the discrete lot-sizes, the integer solution of (P) may 
result in slacks in equation (7.8) and therefore may result in all multipliers 
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of value zero (to satisfy complementary slackness) . Therefore, the multipliers 
Xt for the next iteration were obtained from the LP relaxation of (P). The 
problem is then solved iteratively until the bounds converge. Note that the 
bounds are not guaranteed to converge as there may be a duality gap due to 
discrete nature of the problem. 

7.3.2 Restricting the Problem 

It is comparatively easier to solve the problem when xu has no discontinuous 
discrete integer values. Thus, with the assumption Li := Ei Vz G SKU 2 , 
binary variables zu and equations (7.5) and (7.7) can be eliminated from 
the formulation. Restricting a given problem instance increases the lot-sizes 
for all products in SKU 2 , thereby reducing the set of feasible solutions. As 
we could not find solutions to the unrestricted problem with Cplex, we 
used restricted models for all experiments with IP branch-and-bound. The 
restricted problem is a sub-class of the original problem. 



7.4 Experimental Results 

The experimental results reported in this section are based on a study of real 
data for 190 items and 52 weeks provided by a client of i2 Technologies^ from 
the process industry. The OIP model resulting from the given data is large: 
7520 finite domain variables (average domain size 10) and 3047 constraints 
(average number of variables 30, 1525 constraints soft). 

To summarize the experimental results from the viewpoint of the client, 
what did the study achieve? (i) It found a solution which is provably within 
1.4% of the optimal total cost for constant labor (two shifts), which (ii) shows 
that substantially cutting down cost requires reducing labor, (iii) It showed 
that labor can be reduced to one shift in up to 25 weeks with over 15% 
potential savings of total cost (or USD 1.9 million). 

7.4.1 Comparison of Results 

Table 7.3 reports the best solutions found by Cplex and Wsat(oip) in 
limited time and for different labor profiles. The table divides horizontally 
and vertically, distinguishing the original from the restricted model and the 
fixed-capacity from the varied-capacity case. With respect to overall quality, 
the best solutions among all profiles obtained from both methods are similar 
(Wsat(oip) leading by less than 2%, or USD 223,295). In the experiments, 
the runtime of Wsat(oip) was limited to 10 minutes, Cplex was allowed 30 
minutes for optimization and was cut-off after 60 minutes in case no feasible 
solution was found. All experiments were performed on a Sun Sparc Ultra II. 

^ A leading provider of supply chain scheduling systems. 
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Table 7.3. Computational results (Dollar costs) of Wsat(oip) and Cplex. 
The restricted model forces Li := Ei. 





real problem 


restricted problem 


cost 


Wsat(oip) 


Cplex 


Wsat(oip) 


profile 


fixed capacity, two shifts (230K) 


labor 


11,960,000 


11,960,000 


11,960,000 


inventory 


1,023,106 


1,120,680 


1,040,373 


total 


12,983,106 


13,080,680 


13,000,373 


profile 


one shift [28,52] 


[32,51] 


[29,52] 


labor 


9,085,000 


9,660,000 


9,200,000 


inventory 


1,961,049 


1,609,344 


2,003,884 


total 


11,046,049 


11,269,344 


11,203,884 



Run-times were kept short because many labor profiles had to be examined 
to find solutions of good overall quality. 

Figure 7.2 and 7.3 visualize the experiments across different labor profiles 
for both Wsat(oip) and Cplex. Each impulse represents the total cost of 
the best solution found at one labor profile (start /size coordinates correspond 
to profiles [start, start-l-size— 1], the vertical axis is overall cost). The right 
edge of the triangle reflects the fact that the size of the one-shift period must 
decrease as week 52 is approached, because the planning horizon is finite. 

On the restricted model, Cplex could not And a solution with more than 
21 one-shift periods in the given time while Wsat(oip) was able to solve a 
problem with 25 one-shift periods. In general, Cplex had difficulties to And 
feasible solutions as the labor constraints were tightened. 

Of 115 profiles solved by Wsat(oip), Cplex 5.0 only solved 66 pro- 
files (57%) within the given time limit. For the profiles that could be solved 
with both methods, Wsat(oip) found better solutions in 41 cases; Cplex 
found better solutions in 25 cases, despite the fact that it was applied to 
the restricted model. In the cases where Wsat(oip) [Cplex ] was better, on 
average it improved over Cplex [Wsat(oip) ] by 2.8% [1.4%] with respect 
to pure inventory cost. 



Parameters. Cplex was run with standard parameter settings. Through- 
out the experiments with Wsat(oip), the following parameters were used: 
Initial production was set to zero {pzero = 1); smd a number of 10 tries were 
performed, each with lOOK moves. Allowed variable triggers were limited to 
2 steps up or down the current variable value. Hard constraints were repaired 
with high priority {phard = 0-9). Random moves appeared to deteriorate the 
solution quality, therefore we set Pnoise = 0. A long tabu tenure appeared to 
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Figure 7.2. Solutions 
for various labor pro- 
files (costs in USD). 
Wsat(oip) on the over- 
constrained IP model, 
lOmins per profile. 



Figure 7.3. Cplex 5.0 
for various labor pro- 
files, given the restricted 
MILP model, 30-60mins 
per profile. 



be important to find feasible solutions for problems with very tight capacity 
{t = 100). Constraint weights were critical to obtain good feasible solutions 
and were assigned statically: The hard NOH constraints were weighted with 
a large number, expressing a preference to keep NOH constraints satisfied. 
In contrast, CAP constraints were weighted below 1.0 so that temporarily 
violating them during the search was encouraged. 



Influence of Restricted Model. To better understand the influence of 
the restricted model on the solutions obtained by Cplex, we further exper- 
imented with Wsat(oip) on the restricted model, the results of which are 
plotted in Figures 7.4 and 7.5. Interestingly, Wsat(oip) could still solve 100 
of the 115 profiles given the restricted model. Crosses indicate those labor 
profiles for which feasible solutions exist to the restricted model that Cplex 
could not find within a time bound of Ih. 
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Figure 7.4. Perfor- 
mance of Wsat(oip) 
for the restricted OIP 
model. 







Figure 7.5. Again, 
Cplex on the restricted 
MILP model. Crosses 
indicate profiles that 
could be solved by 
Wsat(oip). 



7.4.2 Lower Bounds 

To assess the quality of the solutions, we applied bound reasoning based on 
Lagrangean relaxation as described above. We used a relaxed labor profile of 
constant 300K, which is over two shifts per week and therefore an unrealistic 
problem. For a precise estimate of the solution quality, Table 7.4 reports 
pure inventory costs based on this profile for the different methods. Using 
Lagrangean decomposition, we found solutions to the relaxed labor profile, 
but unfortunately could not find solutions for realistic capacity constraints. 
Table 7.4 also indicates that Wsat(oip) is still considerably away from the 
best Lagrangean relaxation based solution (3.4% of inventory costs). With 
respect to the overall cost of this profile, the difference vanishes (0.2%). The 
reported lower bound is valid also for the original problem with constant two- 
shift labor, because the 300K-problem is a relaxation of the original problem. 




7.5 Conclusions 



105 



Table 7.4. Solutions (inventory cost) based on a fixed-capacity labor profile 
of 300K in all weeks. 



Solution/bound {Tt = 300K) 


type 


value 


Best IP solution 


restricted 


986,780 


Best solution from Wsat(oip) 


restricted 


973,834 


Best solution from Wsat(oip) 


original 


942,511 


Best Lagrangean solution 


original 


911,960 


Best valid lower bound 


original 


839,875 



7.5 Conclusions 

We have studied a real-world capacitated lot-sizing problem (CLSP) from the 
process industry. Because the problem includes discrete lot-size requirements 
not reported in the CLSP literature, existing domain-specific methods are not 
directly applicable. We approached the problem with Wsat(oip), and exper- 
imentally compared the results to a commercial mixed integer programming 
solver, Cplex. 

The empirical results are promising: Integer local search can solve a CLSP 
with discrete lot-sizes of which a commercial MIP solver can only solve a sub- 
class. In terms of robustness, Wsat(oip) is superior to Cplex on the given 
data, in particular as the capacity constraints are tightened. The ILS model 
is simpler than the MIP model, and with respect to solution quality, the 
techniques are on par. 
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“As fast as computers have become, they’d never be able to solve today’s 
complex business problems without advances in algorithms . . . The future, in 
fact, will be full of algorithms. They are moving up the complexity chain to 
make entire companies more efficient." 



[USA Today; December 31, 1997] 



The primary goals in this book so far have been twofold: To describe new 
effective algorithms within the integer local search framework and to demon- 
strate their capabilities for practical applications. In this section, we will 
critically examine the limitations of the current methods and subsequently 
suggest an extension to overcome some of the current limitations, and provide 
some ideas for future research. 



8.1 Current Limitations 

The limitations of integer local search as laid out here can be characterized 
in terms of (i) the range of problems not under consideration, and (ii) the 
factors not investigated by the experimental analysis. 

Range of Problems. The limitations with respect to the range of stud- 
ied problems are the following. First, throughout this book, we assume pure 
integer optimization problems. Practical industrial problems, however, often 
contain a continuous component (witness thereof is the ubiquity of the lin- 
ear programming method). The framework of integer local search presented 
herein does not currently address mixed IPs. 

Also, the case studies do not address problems in which the solutions 
are required to adhere to some intricate and dominating structure, such as 
traveling salesman problems or job-shop scheduling. Such problems require 
solution structure like tours in a traveling salesman problem or schedules in 
job-shop scheduling, which are difficult to maintain by local repairs on the 
level of variable assignments. For those problems, a local search in another 
search space (e. g. swapping cities on a tour or jobs on a critical path in a 
schedule) turns out to be more suitable. Alternatively if it is easy to obtain 
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solution structure constructively, combining a greedy constructive heuristic 
with local moves in an abstract priority space can be very effective, as pro- 
posed in the framework of “Abstract Local Search” by Crawford, Dalai and 
Walser [36]. While the greedy heuristic can incorporate domain knowledge, 
the abstract moves change decisions in a suitable priority space ( “to schedule 
task A earlier increase its priority”). 

Finally, with exception of one problem class (capacitated production plan- 
ning), the problems under consideration happen to contain mainly 0-1 coef- 
ficients. In fact, for the basic version of Wsat(oip), larger coefficients can 
impose a problem since the score gradient favors variables with large coeffi- 
cients. Section 8.2 below suggests a way to extend the basic scoring scheme 
to handle large coefficients. 

Experimental Analysis. Several limitations of the experimental analysis 
should be discussed that translate directly into suggestions for future re- 
search. First, throughout this book, performance has been studied in terms 
of time to optimal solutions and quality of solutions obtained in limited time. 
Generally, both measures of an optimization algorithm are derived from its 
underlying convergence behavior. To better understand the tradeoffs between 
time and quality, an investigation of the convergence behavior would be nec- 
essary. 

Second, the proposed integer local search procedures contain several com- 
ponents and parameters, such as tabu search components, tie-breaking rules, 
noise, etc. Although some parameters have been investigated in the experi- 
mental analysis in individual case studies, a more rigorous study would be 
required to assess which algorithmic components and parameters are indeed 
critical for the success of integer local search and which parameter settings 
are optimal. An alternative strategy would be to strive for automatically 
adjusting parameters. 

The case studies have demonstrated that OIP models capture the struc- 
ture of many realistic problems via soft constraints, and this structure can 
be exploited by integer local search. Additionally, it would be interesting to 
consider OIP encodings of standard benchmark problems, e. g. problems from 
OR-library [16]. Even though many such benchmarks are randomly generated 
and limited with respect to their constraints, using such standard benchmarks 
would allow to compare domain-independent local search to domain-specific 
optimization heuristics from the literature, as a supplement to the compari- 
son study using general-purpose frameworks. 



8.2 An Alternative Scoring Scheme 

In all of the case studies, the score as defined in (3.6) has been employed, lead- 
ing to a satisfactory operational performance of Wsat(oip). With exception 
to production planning, most of the arising coefficients happened be limited 
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to { — 1, 0, 1}. Preliminary experiments indicate that instances which contain 
larger coefficients may be more difficult for Wsat(oip). For instance, we no- 
tice that the production planning problem requires manual weight setting. 
Further, some problems studied in the literature (e. g. [30]) contain larger co- 
efficients and are not satisfactorily solved. As will be shown, this limitation 
is at least partly due to the applied standard score. Further, we will see that 
the standard score lacks motivation from a geometrical viewpoint. We will 
therefore propose an alternative scoring scheme next. 

Let us recall the example from Section 3.4 (illustrated in Figure 3.7), 



(A) 


9a;i -h 5x2 


> 45 


(B) 


Xi X2 


> 6 


(C) 


8xi -h 5x2 


< 0 (soft) 




Xi,X2 


e {1,2,. ..,5} 



We observe that at point (0,0), the contribution of constraint B to the 
score is 6 while A’s contribution is 45. This is despite the fact that the 
geometric distance of point (0,0) to A and B is almost identical, and the 
Manhattan distance of (0,0) to A is 5, i.e. even smaller than the distance to 
B of 6. 

This observation motivates reconsidering the distance function, and using 
the geometric Euclidean distance. From analytic geometry (e. g. [97]), we 
know that every inequality defines a half-space. In order to compute the 
distance of a point p to a half-space ax > 6 (if p lies outside), we compute 
its distance to the corresponding hyper-plane ax = b. 

To compute the distance, note that a hyper-plane can be represented in 
Hessian normal form, (l/jaj)((a, x) — 6) = 0, where a is a normal vector of 
the hyper-plane, (x, y) defines the scalar product XiPi, and |aj := a/ ( a, a) . 
The distance of a point p to the hyper-plane is then computed [97] as 
(l/|a|) |(a,p) -6|. 

Hence, the Euclidean distance of a point p from the half-space defined by 
the constraint ax > b can be computed as 



||6-ap| 



( 8 . 1 ) 



where jj.jj is defined as usual (OIP), and the resulting distance is 0 if p lies 
inside the half-space.^ 

Surprisingly, we observe that for constraint ax > 6, (8.1) is exactly the 
original score weighted by l/|aj. That is. Euclidean scoring amounts to a 
weighting scheme that can be computed statically as a preprocessing of the 
constraints. 

To preserve equality between the score of a feasible solution and its ob- 
jective function value, we propose to leave the soft constraints unweighted. 

^ Note that ap is the scalar product since a is a row vector. 
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Hence, the Euclidean score for an OIP ((a^), b, C, d, D) shall be defined as 
follows 



score{x) = Whard ■ lib - Ax\\x‘ + ||Cx- d||, (8.2) 

where the weight vector is defined as A® = l/|ai| for constraint a^x > 
and ||.|1 a is defined as usual (3.6). Additionally, Whard is a weight on the hard 
constraints. 

For confined OIPs, one geometrically motivated way to choose Whard is to 
base it on the corresponding objective function: If the soft constraint violation 
is IICx — d|ji and we write this as a classic objective function cx — d := 
IICx— d|| 1 (which is possible as the OIP is confined), then Whard can be chosen 
as |c| in order to balance the overall violation of hard and soft constraints. 

Example. In example (3.12), the Euclidean score results in a weighting 
scheme of w^ard = + 5^ = 9.43 due to the soft constraint. Further, 

the weights are A (4 = l/-\/9^ + 5^, and Ag = l/Vl + 1- In the trajectory ex- 
ample, the same optimal trajectory is obtained as illustrated in Figure 3.8(c), 
which was obtained for a simpler, manually generated weighting scheme. 

Preliminary experiments. Two preliminary experiments can be reported with 
Euclidean scoring. First, in the production planning case study. Euclidean 
scoring automatically leads to a similar weighting scheme and to similar ex- 
perimental results as the scheme obtained in a tedious process of manual 
adjustment. 

Second, to probe the effectiveness of Euclidean scoring, we tested it on 
a benchmark problem from generalized assignment (GAP) from OR-library 
[16] (GAP has been described in Section 6.2.2). We considered one prob- 
lem instance from the set of ‘large-sized’ instances, referenced by G-1 (500 
variables, 110 constraints) and described by Ghu and Beasley [30]. 

In the experiment, we employed deterministic rounding of the linear re- 
laxation to initialize the 0-1 variables. Using standard scoring, Wsat(oip) 
could not find a feasible solution to problem G-1. When applying Euclidean 
scoring, an integer solution (with cost 1934) was usually found within a few 
seconds by Wsat(oip), which is close to the optimal IP solution (1931) and 
to the LP optimum (1923.975). 

It took significantly longer (many restarts) to find the IP optimum using 
this technique. We expect that the technique of dynamic search space reduc- 
tion [9, 5] (described in Section 3.3.3) will improve on this result, considering 
that 341 out of 500 binary variables can be pruned after a few seconds, 
based on the 1934-solution. Moreover, the problem class would provide a 
good test case for leashed local search (Section 3.3.2) since the percentage 
of non-integral solutions of the LP optimum is very small. We are looking 
forward to perform an extensive experimental study of integer local search 
for this problem class. 




8.3 Future Research 111 



8.3 Future Research 



Within the endeavor to find general-purpose heuristics for combinatorial op- 
timization, this work has established a link between local search for proposi- 
tional satisfiability (SAT) and integer optimization. From this perspective, as 
SAT local search strategies are continuously improving, the particular strat- 
egy is insignificant compared to the possibilities that arise due to the link, 
encouraged by the empirical results of this first generalization. SAT can be 
used as a test-bed to obtain better core algorithms while integer encodings 
can leverage their applicability. 

We discuss some of the future paths that appear most promising to pursue 
in order to further improve and extend the proposed methods. 

Incorporating Meta-heuristics and Learning. A variety of meta-heuristic tech- 
niques have been proposed for combinatorial optimization, and sophisticated 
strategies have been presented for particular problem classes. In particu- 
lar genetic algorithms and tabu search [61] offer a variety of strategies that 
would be immediately applicable in an integer local search framework, some 
of which are very likely to enhance the proposed strategies (e. g. tabu search 
intensification and more complex diversification rules, or genetic crossover). 

A route that we predict will lead to very powerful integer local search 
solvers is the incorporation of learning strategies recently proposed for local 
search. Candidate strategies include reactive search [15, 14] which proposes 
a history-based feedback scheme, or Stage [23] which automatically learns 
evaluation (scoring) functions for combinatorial optimization. Also, learning 
strategies for constraint- weights [132, 111, 28] or arc- weights [140] would be 
interesting to integrate in the Wsat(oip) framework for further performance 
improvements . 

Extensions of Integer Local Search. An interesting path to investigate is the 
connection between iterative repair and mathematical programming, which 
is a largely unexplored area. There will be a need to address the combination 
of integer local search with optimization strategies for continuous variables. 
Possibly, integer local search might be combined with other heuristics for 
integer programming to achieve this goal. Also, the formal incorporation of 
maximization functions into OIP will be a task to address. 

With respect to the supply of constraints, we expect to have more ex- 
pressive constraints available soon to extend the current expressivity of OIP. 
In several practically relevant cases, this can be done without the need to 
change the current repair strategy. For instance, within ILP it is difficult to 
express in the constraint minseq{k, [xi, . . . , x„]) = I, which requires that if 
one of the variables Xi is assigned to I, then it must be part of a subsequence 
of at least k variables that are all assigned to 1. This constraint can naturally 
be handled by the current repair mechanism. 
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Hierarchical Factoring. In many real-life situations, a given optimization 
problem is embedded into a larger context. Often, some high-level decisions 
dominate many low-level decisions because they have strong ramifications 
and trigger which low-level decisions need to be made. For example, in the 
Progressive Party Problem, deciding if a given boat is host or guest domi- 
nates the decision which boat a guest visits in a given time period. In this 
sense, there is a decision hierarchy in many real-life optimization problems 
that needs to be accounted for. Tree search approaches naturally account for 
such decision hierarchies by simply first making the high-level decision. In 
contrast, all current local search frameworks lack any support for a decision 
hierarchy, and it would be an interesting task to develop repair strategies 
that take decision hierarchies into account. 

Towards a Local Search Based Constraint Solver. The class of integer linear 
programs covers a wide range of practically important problems and provides 
a good starting point for general-purpose heuristics. Nevertheless, some prob- 
lems exhibit a more complex structure and need more expressive constraint 
representations. In contrast to integer programming frameworks that rely on 
linear relaxations, local search is not limited with respect to the underlying 
constraint systems. Hence, local search strategies Wsat(T) are likely to ap- 
pear that can handle more complex constraint systems X, or even constraints 
from very different domains. 

The current two-stage control strategy is (i) select a constraint c for repair, 
(ii) select a partial repair for constraint c. We expect that this control strategy 
allows for integrating a variety of more complex constraints. For instance, 
symbolic constraints from finite domain constraint programming (e. g. all- 
different) or constraints that address more complex structure (e.g. traveling 
salesman tours) will need to be integrated. Different constraints will require 
different strategies of local repair. The challenge of mixed constraint systems 
will be to extend the two-stage control strategy into a mature architecture 
that integrates different local neighborhoods and effectively control execution. 

In addition to handling problems that are larger or more constrained^ 
the challenge for local search will thus be to handle more complex problem 
structure. For instance, optimally planning the manufacturing process of a 
set of items by sequencing a number of ordered tasks that allocate different 
resources, while minimizing lateness and inventory costs. We predict that the 
complex structure of such real-world scenarios will be an important measure 
of the next generation of local search architectures. 



8.4 Conclusions 

In this monograph we have presented a new effective approach to domain- 
independent integer optimization based on generalizing local search for propo- 
sitional satisfiability. The approach is applicable to a wide variety of combina- 
torial optimization problems that arise in practical applications. It operates 
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on an algebraic representation similar to integer linear programs and is thus 
flexible and can directly be applied to realistic problem encodings. 

In this integer local search framework, a combinatorial optimization or 
constraint problem is stated by an encoding with hard and soft linear con- 
straints over finite domain variables, called an over-constrained integer pro- 
gram. The structure of this representation lends itself well to iterative repair 
approaches since it encodes the optimization objectives by many competing 
soft constraints instead using of a monolithic objective function. With re- 
spect to expressivity, we have shown that this representation is a special-case 
of integer linear programs. 

While the local search strategy that we have presented is simple, we have 
empirically demonstrated its efficiency, scalability and robustness in a vari- 
ety of case studies on realistic integer optimization problems. The problems 
either stem from the recent literature, from operating applications, or from 
industrial cooperation. 

We have experimentally evaluated the described methods in comparisons 
with the literature and with general-purpose optimization strategies. The 
results show that integer local search outperforms or competes with state- 
of-the-art integer programming (IP) branch-and-bound and constraint pro- 
gramming (CP) approaches for the problems under consideration, in finding 
feasible or near-optimal solutions in limited time. The presented method, 
Wsat(oip), is arguably general-purpose because neither integer program- 
ming branch-and-bound nor finite domain constraint programming can cur- 
rently solve the range of problems that have all been solved with integer local 
search in our case studies. 

A drawback of all current local search strategies is their incompleteness, 
that is their inability to prove infeasibility of an input problem or the qual- 
ity of the achieved solutions. To partially overcome this drawback, we have 
discussed several effective combinations with linear programming for lower 
bounding, initialization by rounding, search space reduction and feasibility 
testing. We believe that the iterative repair strategy of integer local search of- 
fers many opportunities for improvements to the core strategy and for further 
generalization to more expressive constraint systems, making it applicable to 
structurally yet more complex problems in the future. 
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The following AMPL model describes the full set of constraints [115, 141] 
of the ACC 97/98 basketball scheduling problem which was investigated in 
Section 5.2. 



## 

## AMPL Model of the Atlantic Coast Competition 1997/98 
## (Basketball/Sports scheduling) 

## 1997-99 J.P.Walser, Programming Systems Lab, UdS 

## 

## Parameters 
## 



set Teams ordered 

set Places ordered 
set Rounds ordered 
set Weekdays ordered 
set Weekends ordered 
set February ordered 
set Mirror 

par am Final 

param GameQualityWeekend 
param GameQualityWeekday 



{ ’Clem’, ’Duke’, ’FSU’ , ’GT’, ’UMD’, 
’UNC’, ’NCSt’, ’UVA’, ’Wake’ }; 

Teams union { ’Bye’ }; 

{ 1 . . 2*card(Teams) I; 

{ 1 .. last (Rounds) by 2 I; 

{ 2 .. last (Rounds) by 2 1; 

{ 11. .18} ; 

{ (1,8), (2,9), (3,12), (4,13), (5,14), 
(6,15), (7,16), (10,17), (11,18) }; 

:= last (Rounds) ; 

{Teams, Teams}; 

{Teams, Teams}; 



## Variables: Pairings 

## T[i,j,t]=l iff team i plays at place j in round t 
var T {Teams, Places, Rounds} binary; 



## 

## DDRR constraints 

## 

subject to OP {i in Teams, t in Rounds}: 
sum { j in Places} T[i,j,t] = 1; 

subject to DV {j in Teams, t in Rounds}: 

sum {i in Teams: ioj} T[i,j,t] <= 1; 

subject to CP {i in Teams, j in Teams, t in Rounds: iOj}: 
T[j,j,t] - T[i,j,t] >= 0; 

subject to DRR {i in Teams, j in Teams: ioj}: 
sum {t in Rounds} T[i,j,t] = 1; 
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## 

## Redundamt constraints 
## 

subject to TB {i in Teams}: 

sum {t in Rounds} T[i,’Bye’,t] = 2; 
subject to DB {t in Rounds}: 

sum {i in Teams} T[i,’Bye’,t] = 1; 
subject to HH {t in Rounds}: 

sum {i in Teams} Tfi.i.t] = floor(card(Teams)/2) ; 



## 

## Sequence constraints (Rounds) 

## 

# Treating Bye as Home, no more than 2 Away games 

# in a row: [ #(A)<=2 ] 

subject to SEQl {i in Teams, t in Rounds : 
t <= prev(Final, Rounds, 2)}: 

sum {s in t . .next (t ,Rounds , 2) , o in Teams: oOi} T[i,o,s] <= 2; 



# Treating Bye as Away, no more than 2 Home games 

# in a row: [ #(H)<=2 ] 

subject to SEQ3 {i in Teams, t in Rounds : 
t <= prev(Final, Rounds, 2)}: 
sum {s in t . .next (t ,Rounds ,2)} T[i,i,s] <= 2; 



# Treating Bye as Away, no more than 3 Away games 

# in a row: [ #(BA)<=3 ], (Bye+Away=not (Home) ) 

subject to SEQ2 {i in Teams, t in Rounds : 

t <= prev(Final, Rounds, 3)}: 
sum {s in t . .next (t ,Rounds ,3)} (1-T[i,i,s]) <= 3; 

# Treating Bye as Home, no more thcui 4 Home games 

# in a row: #(BH)<=4 ] 

subject to SEQ4 {i in Teams, t in Rounds : 
t <= prev (Final, Rounds, 4)}: 

sum {s in t . .next (t , Rounds ,4)} (T[i,i,s] + T [i , ’Bye ’ , s] ) <= 4; 



## 

## Sequence constraints (Weekends) 

## 

# Treating Bye as Home, no more than 2 Away games 

# in a row: [ #(A)<=2 ] 

subject to SEQlw {i in Teams, t in Weekends : 

t <= prevdast(Weekends) , Weekends, 2)}: 
sum {s in t . .next (t ,Weekends ,2) , o in Teams: s in Weekends 
and oOi} T[i,o,s] <= 2; 

# Treating Bye as Away, no more than 2 Home games 

# in a row: [ #(H)<=2 ] 

subject to SEQ3w {i in Teams, t in Weekends : 

t <= prevdast(Weekends) , Weekends, 2)}: 
sum {s in t . .next (t , Weekends , 2) : s in Weekends} T[i,i,s] <= 2; 
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# Treating Bye as Away, no more them 3 Away games 

# in a row: [ #(BA)<=3 ], (Bye+Away=not (Home) ) 

subject to SEQ2w {i in Teams, t in Weekends : 

t <= prev (last (Weekends) , Weekends, 3)}-: 
sum {s in t . .next (t , Weekends , 3) : s in Weekends} 
d-T[i,i,s]) <= 3; 

# Treating Bye as Home, no more than 4 Home games 

# in a row: #(BH)<=4 ] 

subject to SEQ4w {i in Teams, t in Weekends : 

t <= prevdast(Weekends) , Weekends, 3)}: 
sum {s in t . .next (t , Weekends , 3) : s in Weekends} 
(T[i,i,s] + T[i, ’Bye’ ,s]) <= 3; 



## 

## Mirror constraints 
## 

subject to MIRl {i in Teams, j in Teams, (s,t) in Mirror: ioj}: 
d-T[i,j,s]) + T[j,i,t] >= 1; 
subject to MIR2 {i in Teams, (s,t) in Mirror}: 

(1-T[i, ’Bye’ ,s] ) + T[i,’Bye’,t] >= 1; 

## 

## ACC Specific Constraints 
## 

# No team finishes AA 
subject to FAA {i in Teams}: 

sum {t in prev(Final,Rounds) . .Final} 

(T[i,i,t] + T[i, ’Bye’ ,t] ) >= 1; 

# Of 9 weekend Rounds, each team plays 4 home, 4 on the road 

# and one bye 

subject to SATl {i in Teams}: 

sum {t in Weekends} T[i,i,t] = 4; 
subject to SAT2 {i in Teams}: 

sum {t in Weekends} T[i,’Bye’,t] = 1; 

# Home or bye at least on two of first five weekends 
subject to FIF {i in Teams}: 

sum {t in Weekends: ord(t) <= 5} 

(T[i,i,t] + T[i, ’Bye’ ,t] ) >= 2; 

## 

## ACC/season specific constraints 
## 

subject to RIV : # Rival matches 



T[’Duke’ 


, ’UNC’ , 


Final] 


+ 


T[’UNC’ , 


’Duke’ , Final] 


+ 


T[’Clem’ 


, ’GT’ , 


Final] 


+ 


T[’GT’ , 


’Clem’ , Final] 


+ 


T[’NCSt’ 


, ’Wake ’ 


, Final] 


+ 


T[’Wake’ 


, ’NCSt’ , Final] 


+ 


T[’UMD’ , 


’UVA’ , 


Final] 


+ 


T[’UVA’ 


, ’UMD’, Final] 


> = 



# Popular matches in Feb 

subject to FEB {(i,j) in {( ’Wake ’, ’UNC’ ),( ’Wake ’, ’Duke ’) , 
(’GT’ , ’UNC’) , (’GT’ , ’Duke’)}}: 
sum {t in February} (T[i,j,t] + T[j,i,t]) >= 1; 




118 A. A Complete AMPL Model for ACC97/98 



# Opponent ordering constraints 

subject to OPDa {i in Teams, t in Rounds: iO’Duke’ and iO’UNC’ 
and t <= prevCFinal, Rounds)} : 
sum {s in t . .next (t , Rounds) } 

(T[i, ’Duke’ ,s] + T[i, ’UNC’ ,s] ) <= 1; 



subject to DPOb {i in Teams, t in Rounds: iO’Duke’ and iO’UNC’ 
and iO’Wake’ and t <= prevCFinal, Rounds, 2)}: 
sum {s in t . .next (t ,Rounds ,2) , o in {’Duke’ , ’UNC’ , ’Wake’}} 
(T[i,o,s] + T[o,i,s]) <= 2; 



## 

## Other idiosyncratic constraints 
## 



set FixGames dimen 4 within {Teams, Places, Rounds, 0..1}; 
subject to FIX {k in 0..1, (i,j,t,k) in FixGames}: T[i,j,t] = k; 



data; 

set FixGames := 

# UNC plays Duke instantiated 



Duke 


UNC 


11 


1 




UNC 


Duke 


18 


1 




Duke 


Bye 


16 


1 




Wake 


Wake 


17 


0 




# as Wake is 


bye in 


Slot 1 


the othe] 


Wake 


Bye 


1 


1 




Clem 


Clem 


1 


1 




FSU 


FSU 


1 


1 




GT 


GT 


1 


1 




# duke 


CEUinot be bye 


here 


either 


Duke 


Duke 


18 


1 




# rest 










FSU 


Bye 


18 


0 




NCSt 


Bye 


18 


0 




UNC 


Bye 


1 


0 




# additions 


[from Trick’s 


revisions 


# Wake 


has a 


bye in 


slot 1 


Euid must 


Wake 


Wake 


18 


1 




Wake 


Bye 


17 


0 





of May 8, 1998] 
end AH 



model; 



subject to IDI2 
subject to IDI3 
subject to IDI5 
subject to IDI6 



T[’UNC’ , ’Clem’ ,2]+T[’Clem’ , ’UNC’ ,2] = 1; 

T [ ’ Clem ’ , ’ Clem ’ , F inal] +T [ ’ Clem ’ , ’ Bye ’ , F inal] = 1 ; 
T[’UMD’ , ’UMD’ ,Final]+T[’UMD’ , ’Bye’ , Final] = 1; 

T [’Wake ’, ’Wake ’ ,Final] +T [’Wake ’, ’Bye ’ ,Final] = 1; 



## 

## From Trick’s revisions of May 9, 1998 
## 



# every team must have an H in the first three slots 
subject to FTH {i in Teams}: 

sum {t in 1..3} T[i,i,t] >= 1; 
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# every team must have an H in the last three slots 
subject to LTH {i in Teams}: 

sum {t in prev(Final,Rormds,2) . .Final} T[i,i,t] >= 1; 



## 

## Optimization Criteria 
## 

## 

## Criterion 1 

## Avoid opening AA (not more than 1 team) 

## use the complement: sum over teams home or bye >= card(Teams) -1 
## 

subject to DAA: sum {i in Teams} 

(T[i,i,l]+T[i, ’Bye’ ,1] 

+ T [i , i , 2] +T [i , ’Bye ’ ,2] ) >= card(Teams) -1 ; 



## 

## Criterion 2 

## Game qualities: A/B/bad Rounds 
## 

## Variables: Slot-Quality 

## Each slot is either an A,B, or bad slot 

## 

var Q {February, 0..2} binary; 

subject to SLDTQ {t in February}: sum {q in 0..2} Q[t,q] = 1; 

data; 

# GameQualityWeekday [H,A]=2 meauis if team H plays 

# home and A visits it is a 2 match (quality A) 

# A-match: 2, B-match: 1 

param GameQualityWeekday : 





Clem Duke 


FSU 


GT UMD UNC NCSt UVA Wake := 


Clem 


0 


0 


0 


0 


0 


1 


0 


0 


0 


Duke 


0 


0 


0 


1 


2 


0 


0 


1 


1 


FSU 


0 


0 


0 


0 


0 


0 


0 


0 


0 


GT 


0 


1 


0 


0 


1 


2 


0 


0 


1 


UMD 


0 


2 


0 


1 


0 


2 


0 


1 


0 


UNC 


1 


2 


0 


1 


1 


0 


0 


0 


0 


NCSt 


0 


1 


0 


0 


0 


1 


0 


0 


1 


UVA 


0 


1 


0 


0 


0 


0 


0 


0 


1 


Wake 


0 


1 


0 


1 


0 


1 


1 


0 


0 ; 



param GameQualityWeekend : 

Clem Duke FSU GT UMD UNC NCSt UVA Wake := 
Clem 000002000 
Duke 000122011 
FSU 000000000 
GT 000010001 
UMD 000000000 
UNC 000110000 
NCSt 010001001 
UVA 010000000 
Wake 010101100; 
model; 
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# link T and Q variables: If a slot is A, there is at 

# least one A or at least two B games 
subject to LTQl -ft in February: t in Weekends}: 

2*Q[t,2] <= sum {v in Teams, h in Teams: vOh} 

GameQualityWeekend[h,v] * T[v,h,t]; 



subject to LTQ2 {t in February: t in Weekdays}: 

2*Q[t,2] <= sum {v in Teams, h in Teams: vOh} 

GameQualityWeekday [h, v] * T[v,h,t]; 



# if a slot is B, there is at least one B game 
subject to LTQ3 {t in February: t in Weekends}: 

l*Q[t,l] <= sum {v in Teams, h in Teams: vOh} 

GameQualityWeekend[h,v] * T[v,h,t]; 



subject to LTQ4 {t in February: t in Weekdays}: 

l*Q[t,l] <= sum {v in Teams, h in Teams: vOh} 

GameQualityWeekday [h,v] * T[v,h,t]; 



# Require 3 A Rounds in February 

subject to MAXARounds : sum {t in February} Q[t,2] >= 3; 

# Require <= 2 bad Rounds in February 

subject to MINBADRounds : sum {t in February} Q[t,0] <= 2; 
## 

## Criterion 3 

## Home/Away/Bye pattern criteria 
## 

set RoundsM3 ordered := { 1 .. last (Rounds) -2 }; 

set RoundsM3We ordered := { 2 .. last (Rounds) -2*2 by 2 }; 

## 

## Variables expressing optimization criteria 



## 


e.g. 


for each team i and round t. 




## 


HB3[i,t] 


=1 if a sequence of at 


least 3 


## 




home games starts for team t in round i 


## 


NOTE: 


"X=l if 


Y" reads "Y -> X" (not 


iff) 


## 










var 


HB3 


{Teams , 


RoundsMS} binary; 




var 


AB3 


{Teams , 


RoundsMS} binary; 




var 


HB3We 


{Teams , 


RoundsMSWe} binary; 




var 


AB3We 


{Teams , 


RoundsMSWe} binary; 





# HB3=1 if, treating Bye as Home, 3 Home games occur in a row 
subject to HB3L {i in Teams, t in RoundsMS}: 

sum {s in t .. next (t , Rounds , 2)} 

(T[i,i,s] + T[i, ’Bye’ ,s] ) <= 2+HB3[i,t]; 

# AB3=1 if, treating Bye as Away, 3 Away games occur in a row 
subject to AB3L {i in Teams, t in RoundsMS} : 

sum {s in t .. next (t , Rounds , 2)} 
d-T[i,i,s]) <= 2+AB3[i,t] ; 
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# Similarly for weekends: 

# HB3=1 if, treating Bye as Home, 3 Home games occur in a row: 
subject to HB3LWe {i in Teams, t in RoundsMSWe]- : 

sum {s in t .. next (t , Weekends , 2) : s in Weekends} 
(T[i,i,s] + T[i, ’Bye’ ,s]) <= 2+HB3We [i,t] ; 



# AB3=1 if, treating Bye as Away, 3 Away games occur in a row: 
subject to AB3LWe {i in Teams, t in RoundsM3We} : 

sum {s in t .. next (t , Weekends ,2) : s in Weekends} 
d-T[i,i,s]) <= 2+AB3We[i,t] ; 



## Formulate constraints on optimization criteria 
subject to HB3LE4: 

sum {i in Teams, t in RoundsM3} HB3[i,t] <= 4; 
subject to AB3LE3: 

sum {i in Teams, t in RoundsM3} AB3[i,t] <= 3; 
subject to HB3WeLE5: 

sum {i in Teams, t in RoundsMSWe} HB3We[i,t] <= 5; 
subject to AB3WeLE4: 

sum {i in Teams, t in RoundsMSWe} AB3We[i,t] <= 4; 
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